tart run: do not remove "Edit" menu as its not present anymore (#946 )

tart pull: retry if we get URLError (#947 )
Allow to specify custom image in benchmarks (#941 )
2026-10-01 19:51:10 +02:00 · 2024-11-18 09:55:55 +01:00 · 2024-11-15 23:14:47 +01:00 · 2024-11-08 16:41:05 +00:00 · 2024-11-07 21:12:30 +00:00 · 2024-11-07 15:20:09 -05:00
59 changed files with 1681 additions and 410 deletions
@@ -3,3 +3,5 @@
 TMPFILE=$(mktemp)
 envsubst < Sources/tart/CI/CI.swift > $TMPFILE
 mv $TMPFILE Sources/tart/CI/CI.swift
+
+/usr/libexec/PlistBuddy -c "Add :CFBundleShortVersionString string ${CIRRUS_TAG}" Resources/Info.plist
@@ -8,11 +8,11 @@ task:
      name: dev-mini
    resources:
      tart-vms: 1
+  build_script:
+    - swift build
  test_script:
    - swift test
  integration_test_script:
-    # Build Tart
-    - swift build
    - codesign --sign - --entitlements Resources/tart-dev.entitlements --force .build/debug/tart
    - export PATH=$(pwd)/.build/arm64-apple-macosx/debug:$PATH
    # Run integration tests
@@ -0,0 +1,64 @@
+# Profiling Tart
+
+## Using `time(1)`
+
+Perhaps, the easiest, but not the most comprehensive way to tell what's going on with Tart is to use the [`time(1)`](https://ss64.com/mac/time.html) command.
+
+In the example below, you will run `tart pull` via `time(1)` to gather generalized CPU, I/O and memory usage metrics:
+
+```shell
+/usr/bin/time -l tart pull ghcr.io/cirruslabs/macos-sequoia-base:latest
+```
+
+**Note:** you need to specify a full path to `time(1)` binary, otherwise the shell's built-in `time` command will be invoked, which doesn't have the `-l` command-line argument.
+
+**Note:** The `-l` command-line argument makes `time(1)` return much more useful information, for example, maximum memory usage.
+
+When running the command above, you'll see the `tart pull` output first as it pulls the image, and then the `time(1)` output, which will be printed once the Tart process finishes:
+
+```
+      172.17 real        10.29 user         8.36 sys
+           353796096  maximum resident set size
+                   0  average shared memory size
+                   0  average unshared data size
+                   0  average unshared stack size
+               23838  page reclaims
+                  35  page faults
+                   0  swaps
+                   0  block input operations
+                   0  block output operations
+                   8  messages sent
+                   8  messages received
+                   0  signals received
+                 146  voluntary context switches
+              222950  involuntary context switches
+         39683070975  instructions retired
+         27562035252  cycles elapsed
+           170920448  peak memory footprint
+```
+
+From the output above, you can tell that `tart pull` spent nearly 90% of time off-CPU (`real` > `user` + `sys`), which means that Tart was mostly waiting for the I/O (be it a network or disk), instead of decompressing disk layers or doing other useful computations.
+
+## Using `xctrace(1)`
+
+[`xctrace(1)`](https://keith.github.io/xcode-man-pages/xctrace.1.html) is a `.trace` format recorder for the [Instruments](https://en.wikipedia.org/wiki/Instruments_(software)) app, which yields much more powerful insights compared to `time(1)`. For example, it can tell which Tart functions spent the most time on the CPU, thus allowing the Tart developers to further optimize these functions.
+
+To use it, make sure that [Xcode](https://developer.apple.com/xcode/resources/) is installed. If you're installing Xcode for the first time on the machine, you'll need to launch it once and click the blue "Install" button. There's no need to choose any platforms except for the macOS.
+
+Once done, you can create a CPU profile of `tart pull`:
+
+```shell
+xctrace record --template "CPU Profiler" --target-stdout - --launch -- /opt/homebrew/bin/tart pull ghcr.io/cirruslabs/macos-sequoia-base:latest
+```
+
+Now that `xctrace(1)` is running, you'll see the `tart pull`-related output first, and once finished, the following line will appear:
+
+```
+Output file saved as: Launch_[...].trace
+```
+
+To view this trace in the Instruments app, simply find this directory in Finder and double-click it. Instruments app will appear:
+
+![](Resources/Instruments.png)
+
+To send this trace, right-click its directory in Finder and choose "Compress [...]". This will result in a similarly named file with a `.zip` at the end, which can now be conveniently sent via email or uploaded.
@@ -1,13 +1,13 @@
 {
-  "originHash" : "6d48639bc0ea02002de0b4f38fe3fce0ddc9d174f2e56180c2ffcbedb7391ef8",
+  "originHash" : "6a15657d8cb1d3e2b447f31aff5b47d6a9655d2262e48ca76476ba525435269b",
  "pins" : [
    {
      "identity" : "antlr4",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/antlr/antlr4",
      "state" : {
-        "branch" : "dev",
-        "revision" : "2703a8516c0fb7fe92db6b9c40e0113f577646d2"
+        "revision" : "cc82115a4e7f53d71d9d905caa2c2dfa4da58899",
+        "version" : "4.13.2"
      }
    },
    {
@@ -24,8 +24,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/groue/Semaphore",
      "state" : {
-        "revision" : "f1c4a0acabeb591068dea6cffdd39660b86dec28",
-        "version" : "0.0.8"
+        "revision" : "2543679282aa6f6c8ecf2138acd613ed20790bc2",
+        "version" : "0.1.0"
      }
    },
    {
@@ -33,8 +33,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/getsentry/sentry-cocoa",
      "state" : {
-        "revision" : "ef4fec9dfb8dd5027b09a4a5c9362feafd118e1a",
-        "version" : "8.24.0"
+        "revision" : "5575af93efb776414f243e93d6af9f6258dc539a",
+        "version" : "8.36.0"
      }
    },
    {
@@ -51,8 +51,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/apple/swift-argument-parser",
      "state" : {
-        "revision" : "46989693916f56d1186bd59ac15124caef896560",
-        "version" : "1.3.1"
+        "revision" : "41982a3656a71c768319979febd796c6fd111d5c",
+        "version" : "1.5.0"
      }
    },
    {
@@ -61,7 +61,7 @@
      "location" : "https://github.com/apple/swift-async-algorithms",
      "state" : {
        "branch" : "main",
-        "revision" : "f05e450f0b909c0e80670a47516c4b9700b9e5da"
+        "revision" : "5c8bd186f48c16af0775972700626f0b74588278"
      }
    },
    {
@@ -78,8 +78,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/apple/swift-collections.git",
      "state" : {
-        "revision" : "f504716c27d2e5d4144fa4794b12129301d17729",
-        "version" : "1.0.3"
+        "revision" : "9bf03ff58ce34478e66aaee630e491823326fd06",
+        "version" : "1.1.3"
      }
    },
    {
@@ -87,8 +87,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/apple/swift-log.git",
      "state" : {
-        "revision" : "e97a6fcb1ab07462881ac165fdbb37f067e205d5",
-        "version" : "1.5.4"
+        "revision" : "9cb486020ebf03bfa5b5df985387a14a98744537",
+        "version" : "1.6.1"
      }
    },
    {
@@ -105,8 +105,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/fumoboy007/swift-retry",
      "state" : {
-        "revision" : "9f133487ffc2ab4539688c29efe57bb1ba31d7b0",
-        "version" : "0.2.3"
+        "revision" : "df9d7b185d2e433147ec0083a73c257e665eea0d",
+        "version" : "0.2.4"
      }
    },
    {
@@ -118,6 +118,15 @@
        "version" : "1.8.0"
      }
    },
+    {
+      "identity" : "swift-xattr",
+      "kind" : "remoteSourceControl",
+      "location" : "https://github.com/jozefizso/swift-xattr",
+      "state" : {
+        "revision" : "f8605af7b3290dbb235fb182ec6e9035d0c8c3ac",
+        "version" : "3.0.0"
+      }
+    },
    {
      "identity" : "swiftdate",
      "kind" : "remoteSourceControl",
@@ -132,8 +141,8 @@
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/nicklockwood/SwiftFormat",
      "state" : {
-        "revision" : "9df3b01f477163b33d5e63c5e2e5b9f946a49c56",
-        "version" : "0.53.6"
+        "revision" : "ab6844edb79a7b88dc6320e6cee0a0db7674dac3",
+        "version" : "0.54.5"
      }
    },
    {
@@ -15,15 +15,16 @@ let package = Package(
    .package(url: "https://github.com/apple/swift-algorithms", from: "1.2.0"),
    .package(url: "https://github.com/apple/swift-async-algorithms", branch: "main"),
    .package(url: "https://github.com/malcommac/SwiftDate", from: "7.0.0"),
-    .package(url: "https://github.com/antlr/antlr4", branch: "dev"),
+    .package(url: "https://github.com/antlr/antlr4", exact: "4.13.2"),
    .package(url: "https://github.com/apple/swift-atomics.git", .upToNextMajor(from: "1.2.0")),
    .package(url: "https://github.com/nicklockwood/SwiftFormat", from: "0.53.6"),
-    .package(url: "https://github.com/getsentry/sentry-cocoa", from: "8.24.0"),
+    .package(url: "https://github.com/getsentry/sentry-cocoa", from: "8.36.0"),
    .package(url: "https://github.com/cfilipov/TextTable", branch: "master"),
    .package(url: "https://github.com/sersoft-gmbh/swift-sysctl.git", from: "1.8.0"),
    .package(url: "https://github.com/orchetect/SwiftRadix", from: "1.3.1"),
    .package(url: "https://github.com/groue/Semaphore", from: "0.0.8"),
    .package(url: "https://github.com/fumoboy007/swift-retry", from: "0.2.3"),
+    .package(url: "https://github.com/jozefizso/swift-xattr", from: "3.0.0"),
  ],
  targets: [
    .executableTarget(name: "tart", dependencies: [
@@ -40,6 +41,7 @@ let package = Package(
      .product(name: "SwiftRadix", package: "SwiftRadix"),
      .product(name: "Semaphore", package: "Semaphore"),
      .product(name: "DMRetry", package: "swift-retry"),
+      .product(name: "XAttr", package: "swift-xattr"),
    ], exclude: [
      "OCI/Reference/Makefile",
      "OCI/Reference/Reference.g4",
@@ -17,23 +17,23 @@ service — a drop-in replacement for the standard GitHub-hosted runners, offeri
  </a>
 </p>

-Many companies are using Tart in their internal setups. Here are a few of them:
+Many companies are using Tart in their internal setups. Here are just a few of them:

 <p align="center">
  <a href="https://atlassian.com/" target=_blank>
    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Atlassian.png" height="65"/>
  </a>
-  <a href="https://krisp.ai/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Krisp.png" height="65"/>
+  <a href="https://www.figma.com/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Figma.png" height="65"/>
  </a>
  <a href="https://mullvad.net/" target=_blank>
    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Mullvad.png" height="65"/>
  </a>
-  <a href="https://shape.dk/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/shape.png" height="65"/>
+  <a href="https://krisp.ai/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Krisp.png" height="65"/>
  </a>
-  <a href="https://suran.com/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Suran.png" height="65"/>
+  <a href="https://testingbot.com/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/TestingBot.png" height="65"/>
  </a>
  <a href="https://symflower.com/" target=_blank>
    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Symflower.png" height="65"/>
@@ -41,18 +41,18 @@ Many companies are using Tart in their internal setups. Here are a few of them:
  <a href="https://transloadit.com/" target=_blank>
    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Transloadit.png" height="65"/>
  </a>
-  <a href="https://uphold.com/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Uphold.png" height="65"/>
+  <a href="https://cirrus-ci.org/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/CirrusCI.png" height="65"/>
  </a>
  <a href="https://www.pitsdatarecovery.net/" target=_blank>
    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/PITSGlobalDataRecoveryServices.png" height="65"/>
  </a>
-  <a href="https://ahrefs.com/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/ahrefs.png" height="65"/>
+  <a href="https://expo.dev/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Expo.png" height="65"/>
  </a>
 </p>

-**Note:** If your company or project is using Tart please consider [adding yourself to the list above](/Resources/Users/HowToAddYourself.md).
+**Note:** If your company or project is using Tart please consider [sharing with the community](https://github.com/cirruslabs/tart/discussions/857).

 <p align="center">
  <a href="https://aws.amazon.com/marketplace/pp/prodview-qczco34wlkdws?utm_source=github&utm_medium=referral" target=_blank>
@@ -2,22 +2,24 @@
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 <plist version="1.0">
 <dict>
-    <key>CFBundleName</key>
-    <string>tart</string>
-    <key>CFBundleIdentifier</key>
-    <string>org.cirruslabs.tart</string>
-    <key>CFBundleExecutable</key>
-    <string>tart</string>
-    <key>LSBackgroundOnly</key>
-    <string>1</string>
-    <key>CFBundleIconFiles</key>
-    <array>
-        <string>AppIcon.png</string>
-    </array>
-    <key>NSAppTransportSecurity</key>
-    <dict>
-        <key>NSAllowsArbitraryLoads</key>
-        <true/>
-    </dict>
+	<key>CFBundleName</key>
+	<string>Tart</string>
+	<key>CFBundleDisplayName</key>
+	<string>Tart</string>
+	<key>CFBundleIdentifier</key>
+	<string>org.cirruslabs.tart</string>
+	<key>CFBundleExecutable</key>
+	<string>tart</string>
+  <key>LSApplicationCategoryType</key>
+  <string>public.app-category.developer-tools</string>
+	<key>CFBundleIconFiles</key>
+	<array>
+		<string>AppIcon.png</string>
+	</array>
+	<key>NSAppTransportSecurity</key>
+	<dict>
+		<key>NSAllowsArbitraryLoads</key>
+		<true/>
+	</dict>
 </dict>
 </plist>
@@ -1,4 +0,0 @@
-If you'd like to highlight your use of Tart, please create a `456px` by `130px` logo and create a PR
-that adds it to `README.md` in alphabetical order. Don't forget to include a small description of your usage pattern.
-
-You can refer to `Background.png` as a base for your logo.
@@ -9,12 +9,10 @@ struct Clone: AsyncParsableCommand {
    Creates a local virtual machine by cloning either a remote or another local virtual machine.

    Due to copy-on-write magic in Apple File System, a cloned VM won't actually claim all the space right away.
-    Only changes to a cloned disk will be written and claim new space. By default, Tart checks available capacity
-    in Tart's home directory and checks if there is enough space for the worst possible scenario: when the whole disk
-    will be modified.
+    Only changes to a cloned disk will be written and claim new space. This also speeds up clones enormously.

-    This behaviour can be disabled by setting TART_NO_AUTO_PRUNE environment variable. This might be helpful
-    for use cases when the original image is very big and a workload is known to only modify a fraction of the cloned disk.
+    By default, Tart checks available capacity in Tart's home directory and tries to reclaim minimum possible storage for the cloned image
+    to fit. This behaviour is called "automatic pruning" and can be disabled by setting TART_NO_AUTO_PRUNE environment variable.
    """
  )

@@ -30,6 +28,9 @@ struct Clone: AsyncParsableCommand {
  @Option(help: "network concurrency to use when pulling a remote VM from the OCI-compatible registry")
  var concurrency: UInt = 4

+  @Flag(help: .hidden)
+  var deduplicate: Bool = false
+
  func validate() throws {
    if newName.contains("/") {
      throw ValidationError("<new-name> should be a local name")
@@ -47,7 +48,7 @@ struct Clone: AsyncParsableCommand {
    if let remoteName = try? RemoteName(sourceName), !ociStorage.exists(remoteName) {
      // Pull the VM in case it's OCI-based and doesn't exist locally yet
      let registry = try Registry(host: remoteName.host, namespace: remoteName.namespace, insecure: insecure)
-      try await ociStorage.pull(remoteName, registry: registry, concurrency: concurrency)
+      try await ociStorage.pull(remoteName, registry: registry, concurrency: concurrency, deduplicate: deduplicate)
    }

    let sourceVM = try VMStorageHelper.open(sourceName)
@@ -7,6 +7,7 @@ fileprivate struct VMInfo: Encodable {
  let Name: String
  let Disk: Int
  let Size: Int
+  let SizeOnDisk: Int
  let Running: Bool
  let State: String
 }
@@ -38,13 +39,13 @@ struct List: AsyncParsableCommand {

    if source == nil || source == "local" {
      infos += sortedInfos(try VMStorageLocal().list().map { (name, vmDir) in
-        try VMInfo(Source: "local", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
+        try VMInfo(Source: "local", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), SizeOnDisk: vmDir.allocatedSizeGB() - vmDir.deduplicatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
      })
    }

    if source == nil || source == "oci" {
      infos += sortedInfos(try VMStorageOCI().list().map { (name, vmDir, _) in
-        try VMInfo(Source: "OCI", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
+        try VMInfo(Source: "OCI", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), SizeOnDisk: vmDir.allocatedSizeGB() - vmDir.deduplicatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
      })
    }

@@ -49,9 +49,10 @@ struct Login: AsyncParsableCommand {
    ])

    if !noValidate {
+      let registry = try Registry(host: host, namespace: "", insecure: insecure,
+                                  credentialsProviders: [credentialsProvider])
+
      do {
-        let registry = try Registry(host: host, namespace: "", insecure: insecure,
-                                    credentialsProviders: [credentialsProvider])
        try await registry.ping()
      } catch {
        throw RuntimeError.InvalidCredentials("invalid credentials: \(error)")
@@ -9,8 +9,8 @@ struct Pull: AsyncParsableCommand {
    Pulls a virtual machine from a remote OCI-compatible registry. Supports authorization via Keychain (see "tart login --help"),
    Docker credential helpers defined in ~/.docker/config.json or via TART_REGISTRY_USERNAME/TART_REGISTRY_PASSWORD environment variables.

-    By default, Tart checks available capacity in Tart's home directory and tries to reclaim minimum possible storage for the remote image to fit via "tart prune".
-    This behaviour can be disabled by setting TART_NO_AUTO_PRUNE environment variable.
+    By default, Tart checks available capacity in Tart's home directory and tries to reclaim minimum possible storage for the remote image
+    to fit. This behaviour is called "automatic pruning" and can be disabled by setting TART_NO_AUTO_PRUNE environment variable.
    """
  )

@@ -23,6 +23,9 @@ struct Pull: AsyncParsableCommand {
  @Option(help: "network concurrency to use when pulling a remote VM from the OCI-compatible registry")
  var concurrency: UInt = 4

+  @Flag(help: .hidden)
+  var deduplicate: Bool = false
+
  func validate() throws {
    if concurrency < 1 {
      throw ValidationError("network concurrency cannot be less than 1")
@@ -43,6 +46,6 @@ struct Pull: AsyncParsableCommand {

    defaultLogger.appendNewLine("pulling \(remoteName)...")

-    try await VMStorageOCI().pull(remoteName, registry: registry, concurrency: concurrency)
+    try await VMStorageOCI().pull(remoteName, registry: registry, concurrency: concurrency, deduplicate: deduplicate)
  }
 }
@@ -15,6 +15,9 @@ struct Push: AsyncParsableCommand {
  @Flag(help: "connect to the OCI registry via insecure HTTP protocol")
  var insecure: Bool = false

+  @Option(help: "network concurrency to use when pushing a local VM to the OCI-compatible registry")
+  var concurrency: UInt = 4
+
  @Option(help: ArgumentHelp("chunk size in MB if registry supports chunked uploads",
                             discussion: """
                             By default monolithic method is used for uploading blobs to the registry but some registries support a more efficient chunked method.
@@ -33,6 +36,10 @@ struct Push: AsyncParsableCommand {
  func run() async throws {
    let ociStorage = VMStorageOCI()
    let localVMDir = try VMStorageHelper.open(localName)
+    let lock = try localVMDir.lock()
+    if try !lock.trylock() {
+      throw RuntimeError.VMIsRunning(localName)
+    }

    // Parse remote names supplied by the user
    let remoteNames = try remoteNames.map{
@@ -73,7 +80,8 @@ struct Push: AsyncParsableCommand {
          registry: registry,
          references: references,
          chunkSizeMb: chunkSize,
-          diskFormat: diskFormat
+          diskFormat: diskFormat,
+          concurrency: concurrency
        )
        // Populate the local cache (if requested)
        if populateCache {
@@ -12,6 +12,39 @@ var vm: VM?
 struct IPNotFound: Error {
 }

+@available(macOS 14, *)
+extension VZDiskSynchronizationMode {
+  public init(_ description: String) throws {
+    switch description {
+    case "none":
+      self = .none
+    case "full":
+      self = .full
+    case "":
+      self = .full
+    default:
+      throw RuntimeError.VMConfigurationError("unsupported disk synchronization mode: \"\(description)\"")
+    }
+  }
+}
+
+extension VZDiskImageSynchronizationMode {
+  public init(_ description: String) throws {
+    switch description {
+    case "none":
+      self = .none
+    case "fsync":
+      self = .fsync
+    case "full":
+      self = .full
+    case "":
+      self = .full
+    default:
+      throw RuntimeError.VMConfigurationError("unsupported disk image synchronization mode: \"\(description)\"")
+    }
+  }
+}
+
 struct Run: AsyncParsableCommand {
  static var configuration = CommandConfiguration(abstract: "Run a VM")

@@ -67,7 +100,7 @@ struct Run: AsyncParsableCommand {
  var vncExperimental: Bool = false

  @Option(help: ArgumentHelp("""
-  Additional disk attachments with an optional read-only specifier\n(e.g. --disk=\"disk.bin\" --disk=\"ubuntu.iso:ro\" --disk=\"/dev/disk0\" --disk "ghcr.io/cirruslabs/xcode:16.0:ro" --disk=\"nbd://localhost:10809/myDisk\")
+  Additional disk attachments with an optional read-only and synchronization options (e.g. --disk="disk.bin", --disk="ubuntu.iso:ro", --disk="/dev/disk0", --disk "ghcr.io/cirruslabs/xcode:16.0:ro" or --disk="nbd://localhost:10809/myDisk:sync=none")
  """, discussion: """
  The disk attachment can be a:

@@ -76,6 +109,12 @@ struct Run: AsyncParsableCommand {
  * remote VM name whose disk will be mounted
  * Network Block Device (NBD) URL

+  Options are comma-separated and are as follows:
+
+  * ro — attach the specified disk in read-only mode instead of the default read-write (e.g. --disk="disk.img:ro")
+
+  * sync=none — disable data synchronization with the permanent storage to increase performance at the cost of a higher chance of data loss (e.g. --disk="disk.img:sync=none")
+
  Learn how to create a disk image using Disk Utility here: https://support.apple.com/en-gb/guide/disk-utility/dskutl11888/mac

  To work with block devices, the easiest way is to modify their permissions (e.g. by using "sudo chown $USER /dev/diskX") or to run the Tart binary as root, which affects locating Tart VMs.
@@ -83,7 +122,7 @@ struct Run: AsyncParsableCommand {
  To work around this pass TART_HOME explicitly:

  sudo TART_HOME="$HOME/.tart" tart run sonoma --disk=/dev/disk0
-  """, valueName: "path[:ro]"))
+  """, valueName: "path[:options]"))
  var disk: [String] = []

  #if arch(arm64)
@@ -117,6 +156,9 @@ struct Run: AsyncParsableCommand {
  """, valueName: "[name:]path[:options]"))
  var dir: [String] = []

+  @Flag(help: ArgumentHelp("Enable nested virtualization if possible"))
+  var nested: Bool = false
+
  @Option(help: ArgumentHelp("""
  Use bridged networking instead of the default shared (NAT) networking \n(e.g. --net-bridged=en0 or --net-bridged=\"Wi-Fi\")
  """, discussion: """
@@ -134,6 +176,20 @@ struct Run: AsyncParsableCommand {
  @Flag(help: ArgumentHelp("Restrict network access to the host-only network"))
  var netHost: Bool = false

+  @Option(help: ArgumentHelp("Set the root disk options (e.g. --root-disk-opts=\"ro\" or --root-disk-opts=\"sync=none\")",
+                             discussion: """
+                             Options are comma-separated and are as follows:
+
+                             * ro — attach the root disk in read-only mode instead of the default read-write (e.g. --root-disk-opts="ro")
+
+                             * sync=none — disable data synchronization with the permanent storage to increase performance at the cost of a higher chance of data loss (e.g. --root-disk-opts="sync=none")
+
+                             * sync=fsync — enable data synchronization with the permanent storage, but don't ensure that it was actually written (e.g. --root-disk-opts="sync=fsync")
+
+                             * sync=full — enable data synchronization with the permanent storage and ensure that it was actually written (e.g. --root-disk-opts="sync=full")
+                             """, valueName: "options"))
+  var rootDiskOpts: String = ""
+
  #if arch(arm64)
    @Flag(help: ArgumentHelp("Disables audio and entropy devices and switches to only Mac-specific input devices.", discussion: "Useful for running a VM that can be suspended via \"tart suspend\"."))
  #endif
@@ -168,6 +224,14 @@ struct Run: AsyncParsableCommand {
      throw ValidationError("--captures-system-keys can only be used with the default VM view")
    }

+    if nested {
+      if #unavailable(macOS 15) {
+        throw ValidationError("Nested virtualization is supported on hosts starting with macOS 15 (Sequia), and later.")
+      } else if !VZGenericPlatformConfiguration.isNestedVirtualizationSupported {
+        throw ValidationError("Nested virtualization is available for Mac with the M3 chip, and later.")
+      }
+    }
+
    let localStorage = VMStorageLocal()
    let vmDir = try localStorage.open(name)
    if try vmDir.state() == .Suspended {
@@ -176,7 +240,7 @@ struct Run: AsyncParsableCommand {

    if suspendable {
      let config = try VMConfig.init(fromURL: vmDir.configURL)
-      if (config.platform is Linux) {
+      if !(config.platform is PlatformSuspendable) {
        throw ValidationError("You can only suspend macOS VMs")
      }
      if dir.count > 0 {
@@ -213,8 +277,6 @@ struct Run: AsyncParsableCommand {
      try Softnet.configureSUIDBitIfNeeded()
    }

-    let additionalDiskAttachments = try additionalDiskAttachments()
-
    var serialPorts: [VZSerialPortConfiguration] = []
    if serial {
      let tty_fd = createPTY()
@@ -233,15 +295,20 @@ struct Run: AsyncParsableCommand {
      serialPorts.append(createSerialPortConfiguration(tty_read!, tty_write!))
    }

+    // Parse root disk options
+    let diskOptions = DiskOptions(rootDiskOpts)
+
    vm = try VM(
      vmDir: vmDir,
      network: userSpecifiedNetwork(vmDir: vmDir) ?? NetworkShared(),
-      additionalStorageDevices: additionalDiskAttachments,
+      additionalStorageDevices: try additionalDiskAttachments(),
      directorySharingDevices: directoryShares() + rosettaDirectoryShare(),
      serialPorts: serialPorts,
      suspendable: suspendable,
+      nested: nested,
      audio: !noAudio,
-      clipboard: !noClipboard
+      clipboard: !noClipboard,
+      sync: VZDiskImageSynchronizationMode(diskOptions.syncModeRaw)
    )

    let vncImpl: VNC? = try {
@@ -291,7 +358,35 @@ struct Run: AsyncParsableCommand {
          }
        #endif

-        try await vm!.start(recovery: recovery, resume: resume)
+        do {
+          try await vm!.start(recovery: recovery, resume: resume)
+        } catch let error as VZError {
+          if error.code == .virtualMachineLimitExceeded {
+            var hint = ""
+
+            do {
+              let runningVMs: [String] = try localStorage.list().compactMap { (name, vmDir) in
+                if try !vmDir.running() {
+                  return nil
+                }
+
+                return name
+              }
+
+              if !runningVMs.isEmpty {
+                let runningVMsJoined = runningVMs.joined(separator: ", ")
+
+                hint = " (other running VMs: \(runningVMsJoined))"
+              }
+            } catch {
+              // we can't provide any hint
+            }
+
+            throw RuntimeError.VirtualMachineLimitExceeded(hint)
+          }
+
+          throw error
+        }

        if let vncImpl = vncImpl {
          let vncURL = try await vncImpl.waitForURL(netBridged: !netBridged.isEmpty)
@@ -377,8 +472,10 @@ struct Run: AsyncParsableCommand {

    let useVNCWithoutGraphics = (vnc || vncExperimental) && !graphics
    if noGraphics || useVNCWithoutGraphics {
-      // enter the main even loop, without bringing up any UI,
-      // and just wait for the VM to exit.
+      // Enter the main event loop without bringing up any UI,
+      // waiting for the VM to exit.
+      NSApplication.shared.setActivationPolicy(.prohibited)
+
      NSApplication.shared.run()
    } else {
      runUI(suspendable, captureSystemKeys)
@@ -449,90 +546,9 @@ struct Run: AsyncParsableCommand {
  }

  func additionalDiskAttachments() throws -> [VZStorageDeviceConfiguration] {
-    var result: [VZStorageDeviceConfiguration] = []
-    let readOnlySuffix = ":ro"
-    let expandedDiskPaths = disk.map { NSString(string:$0).expandingTildeInPath }
-
-    for rawDisk in expandedDiskPaths {
-      let diskReadOnly = rawDisk.hasSuffix(readOnlySuffix)
-      let diskPath = diskReadOnly ? String(rawDisk.prefix(rawDisk.count - readOnlySuffix.count)) : rawDisk
-
-      let diskURL = URL(string: diskPath)
-      if (["nbd", "nbds", "nbd+unix", "nbds+unix"].contains(diskURL?.scheme)) {
-        guard #available(macOS 14, *) else {
-          throw UnsupportedOSError("attaching Network Block Devices", "are")
-        }
-
-        let nbdAttachment = try VZNetworkBlockDeviceStorageDeviceAttachment(
-          url: diskURL!,
-          timeout: 30,
-          isForcedReadOnly: diskReadOnly,
-          synchronizationMode: VZDiskSynchronizationMode.none
-        )
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: nbdAttachment))
-        continue
-      }
-
-      let diskFileURL = URL(fileURLWithPath: diskPath)
-
-      if pathHasMode(diskPath, mode: S_IFBLK) {
-        guard #available(macOS 14, *) else {
-          throw UnsupportedOSError("attaching block devices", "are")
-        }
-
-        let fd = open(diskPath, diskReadOnly ? O_RDONLY : O_RDWR)
-        if fd == -1 {
-          let details = Errno(rawValue: CInt(errno))
-
-          switch details.rawValue {
-          case EBUSY:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "already in use, try umounting it via \"diskutil unmountDisk\" (when the whole disk) or \"diskutil umount\" (when mounting a single partition)")
-          case EACCES:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "permission denied, consider changing the disk's owner using \"sudo chown $USER \(diskFileURL.url.path)\" or run Tart as a superuser (see --disk help for more details on how to do that correctly)")
-          default:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "\(details)")
-          }
-        }
-
-        let blockAttachment = try VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: FileHandle(fileDescriptor: fd, closeOnDealloc: true),
-                                                                           readOnly: diskReadOnly, synchronizationMode: .full)
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: blockAttachment))
-        continue
-      }
-
-      // Support remote VM names in --disk command-line argument
-      if let remoteName = try? RemoteName(diskPath) {
-        let vmDir = try VMStorageOCI().open(remoteName)
-
-        // Unfortunately, VZDiskImageStorageDeviceAttachment does not support
-        // FileHandle, so we can't easily clone the disk, open it and unlink(2)
-        // to simplify the garbage collection, so use an intermediate directory.
-        let clonedDiskURL = try Config().tartTmpDir.appendingPathComponent("run-disk-\(UUID().uuidString)")
-
-        try FileManager.default.copyItem(at: vmDir.diskURL, to: clonedDiskURL)
-
-        let lock = try FileLock(lockURL: clonedDiskURL)
-        try lock.lock()
-
-        let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: clonedDiskURL, readOnly: diskReadOnly)
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment))
-        continue
-      }
-
-      // Error out if the disk is locked by the host (e.g. it was mounted in Finder),
-      // see https://github.com/cirruslabs/tart/issues/323 for more details.
-      if try !diskReadOnly && !FileLock(lockURL: diskFileURL).trylock() {
-        throw RuntimeError.DiskAlreadyInUse("disk \(diskFileURL.url.path) seems to be already in use, unmount it first in Finder")
-      }
-
-      let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(
-        url: diskFileURL,
-        readOnly: diskReadOnly
-      )
-      result.append(VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment))
+    try disk.map {
+      try AdditionalDisk(parseFrom: $0).configuration
    }
-
-    return result
  }

  func directoryShares() throws -> [VZDirectorySharingDeviceConfiguration] {
@@ -615,7 +631,7 @@ struct MainApp: App {
  static var suspendable: Bool = false
  static var capturesSystemKeys: Bool = false

-  @NSApplicationDelegateAdaptor private var appDelegate: MinimalMenuAppDelegate
+  @NSApplicationDelegateAdaptor private var appDelegate: AppDelegate

  var body: some Scene {
    WindowGroup(vm!.name) {
@@ -671,18 +687,7 @@ struct MainApp: App {
  }
 }

-// The only way to fully remove Edit menu item.
-class MinimalMenuAppDelegate: NSObject, NSApplicationDelegate, ObservableObject {
-  let indexOfEditMenu = 2
-
-  func applicationDidFinishLaunching(_ : Notification) {
-    NSApplication.shared.mainMenu?.removeItem(at: indexOfEditMenu)
-
-    let nsApp = NSApplication.shared
-    nsApp.setActivationPolicy(.regular)
-    nsApp.activate(ignoringOtherApps: true)
-  }
-
+class AppDelegate: NSObject, NSApplicationDelegate, ObservableObject {
  func applicationShouldTerminate(_ sender: NSApplication) -> NSApplication.TerminateReply {
    if (kill(getpid(), MainApp.suspendable ? SIGUSR1 : SIGINT) == 0) {
      return .terminateLater
@@ -752,6 +757,134 @@ struct VMView: NSViewRepresentable {
  }
 }

+struct AdditionalDisk {
+  let configuration: VZStorageDeviceConfiguration
+
+  init(parseFrom: String) throws {
+    let (diskPath, readOnly, syncModeRaw) = Self.parseOptions(parseFrom)
+
+    self.configuration = try Self.craft(diskPath, readOnly: readOnly, syncModeRaw: syncModeRaw)
+  }
+
+  static func craft(_ diskPath: String, readOnly diskReadOnly: Bool, syncModeRaw: String) throws -> VZStorageDeviceConfiguration {
+    let diskURL = URL(string: diskPath)
+
+    if (["nbd", "nbds", "nbd+unix", "nbds+unix"].contains(diskURL?.scheme)) {
+      guard #available(macOS 14, *) else {
+        throw UnsupportedOSError("attaching Network Block Devices", "are")
+      }
+
+      let nbdAttachment = try VZNetworkBlockDeviceStorageDeviceAttachment(
+        url: diskURL!,
+        timeout: 30,
+        isForcedReadOnly: diskReadOnly,
+        synchronizationMode: try VZDiskSynchronizationMode(syncModeRaw)
+      )
+
+      return VZVirtioBlockDeviceConfiguration(attachment: nbdAttachment)
+    }
+
+    // Expand the tilde (~) since at this point we're dealing with a local path,
+    // and "expandingTildeInPath" seems to corrupt the remote URLs like nbd://
+    let diskPath = NSString(string: diskPath).expandingTildeInPath
+
+    let diskFileURL = URL(fileURLWithPath: diskPath)
+
+    if pathHasMode(diskPath, mode: S_IFBLK) {
+      guard #available(macOS 14, *) else {
+        throw UnsupportedOSError("attaching block devices", "are")
+      }
+
+      let fd = open(diskPath, diskReadOnly ? O_RDONLY : O_RDWR)
+      if fd == -1 {
+        let details = Errno(rawValue: CInt(errno))
+
+        switch details.rawValue {
+        case EBUSY:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "already in use, try umounting it via \"diskutil unmountDisk\" (when the whole disk) or \"diskutil umount\" (when mounting a single partition)")
+        case EACCES:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "permission denied, consider changing the disk's owner using \"sudo chown $USER \(diskFileURL.url.path)\" or run Tart as a superuser (see --disk help for more details on how to do that correctly)")
+        default:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "\(details)")
+        }
+      }
+
+      let blockAttachment = try VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: FileHandle(fileDescriptor: fd, closeOnDealloc: true),
+                                                                         readOnly: diskReadOnly, synchronizationMode: try VZDiskSynchronizationMode(syncModeRaw))
+
+      return VZVirtioBlockDeviceConfiguration(attachment: blockAttachment)
+    }
+
+    // Support remote VM names in --disk command-line argument
+    if let remoteName = try? RemoteName(diskPath) {
+      let vmDir = try VMStorageOCI().open(remoteName)
+
+      // Unfortunately, VZDiskImageStorageDeviceAttachment does not support
+      // FileHandle, so we can't easily clone the disk, open it and unlink(2)
+      // to simplify the garbage collection, so use an intermediate directory.
+      let clonedDiskURL = try Config().tartTmpDir.appendingPathComponent("run-disk-\(UUID().uuidString)")
+
+      try FileManager.default.copyItem(at: vmDir.diskURL, to: clonedDiskURL)
+
+      let lock = try FileLock(lockURL: clonedDiskURL)
+      try lock.lock()
+
+      let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: clonedDiskURL, readOnly: diskReadOnly)
+
+      return VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment)
+    }
+
+    // Error out if the disk is locked by the host (e.g. it was mounted in Finder),
+    // see https://github.com/cirruslabs/tart/issues/323 for more details.
+    if try !diskReadOnly && !FileLock(lockURL: diskFileURL).trylock() {
+      throw RuntimeError.DiskAlreadyInUse("disk \(diskFileURL.url.path) seems to be already in use, unmount it first in Finder")
+    }
+
+    let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(
+      url: diskFileURL,
+      readOnly: diskReadOnly,
+      cachingMode: .automatic,
+      synchronizationMode: try VZDiskImageSynchronizationMode(syncModeRaw)
+    )
+
+    return VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment)
+  }
+
+  static func parseOptions(_ parseFrom: String) -> (String, Bool, String) {
+    var arguments = parseFrom.split(separator: ":")
+
+    let options = DiskOptions(String(arguments.last!))
+    if options.foundAtLeastOneOption {
+      arguments.removeLast()
+    }
+
+    return (arguments.joined(separator: ":"), options.readOnly, options.syncModeRaw)
+  }
+}
+
+struct DiskOptions {
+  var readOnly: Bool = false
+  var syncModeRaw: String = ""
+  var foundAtLeastOneOption: Bool = false
+
+  init(_ parseFrom: String) {
+    let options = parseFrom.split(separator: ",")
+
+    for option in options {
+      switch true {
+      case option == "ro":
+        self.readOnly = true
+        self.foundAtLeastOneOption = true
+      case option.hasPrefix("sync="):
+        self.syncModeRaw = String(option.dropFirst("sync=".count))
+        self.foundAtLeastOneOption = true
+      default:
+        continue
+      }
+    }
+  }
+}
+
 struct DirectoryShare {
  let name: String?
  let path: URL
@@ -39,9 +39,10 @@ class DockerConfigCredentialsProvider: CredentialsProvider {
    inPipe.fileHandleForWriting.write("\(host)\n".data(using: .utf8)!)
    inPipe.fileHandleForWriting.closeFile()

+    let outputData = try outPipe.fileHandleForReading.readToEnd()
+
    process.waitUntilExit()

-    let outputData = try outPipe.fileHandleForReading.readToEnd()
    if !(process.terminationReason == .exit && process.terminationStatus == 0) {
      if let outputData = outputData {
        print(String(decoding: outputData, as: UTF8.self))
@@ -1,10 +1,25 @@
 import Foundation

 struct LocalLayerCache {
+  struct DigestInfo {
+    let range: Range<Data.Index>
+    let compressedDigest: String
+    let uncompressedContentDigest: String?
+  }
+
+  let name: String
+  let deduplicatedBytes: UInt64
+  let diskURL: URL
+
  private let mappedDisk: Data
-  private var digestToRange: [String : Range<Data.Index>] = [:]
+  private var digestToRange: [String: DigestInfo] = [:]
+  private var offsetToRange: [UInt64: DigestInfo] = [:]
+
+  init?(_ name: String, _ deduplicatedBytes: UInt64, _ diskURL: URL, _ manifest: OCIManifest) throws {
+    self.name = name
+    self.deduplicatedBytes = deduplicatedBytes
+    self.diskURL = diskURL

-  init?(_ diskURL: URL, _ manifest: OCIManifest) throws {
    // mmap(2) the disk that contains the layers from the manifest
    self.mappedDisk = try Data(contentsOf: diskURL, options: [.alwaysMapped])

@@ -16,17 +31,27 @@ struct LocalLayerCache {
        return nil
      }

-      self.digestToRange[layer.digest] = Int(offset)..<Int(offset+uncompressedSize)
+      let info = DigestInfo(
+        range: Int(offset)..<Int(offset + uncompressedSize),
+        compressedDigest: layer.digest,
+        uncompressedContentDigest: layer.uncompressedContentDigest()!
+      )
+      self.digestToRange[layer.digest] = info
+      self.offsetToRange[offset] = info

      offset += uncompressedSize
    }
  }

-  func find(_ digest: String) -> Data? {
-    guard let foundRange = self.digestToRange[digest] else {
-      return nil
+  func findInfo(digest: String, offsetHint: UInt64) -> DigestInfo? {
+    // Layers can have the same digests, for example, empty ones. Let's use the offset hint to make a better guess.
+    if let info = self.offsetToRange[offsetHint], info.compressedDigest == digest {
+      return info
    }
+    return self.digestToRange[digest]
+  }

-    return self.mappedDisk.subdata(in: foundRange)
+  func subdata(_ range: Range<Data.Index>) -> Data {
+    return self.mappedDisk.subdata(in: range)
  }
 }
@@ -52,6 +52,11 @@ struct ARPCache {
    process.standardInput = FileHandle.nullDevice

    try process.run()
+
+    guard let arpCommandOutput = try pipe.fileHandleForReading.readToEnd() else {
+      throw ARPCommandYieldedInvalidOutputError(explanation: "empty output")
+    }
+
    process.waitUntilExit()

    if !(process.terminationReason == .exit && process.terminationStatus == 0) {
@@ -60,10 +65,6 @@ struct ARPCache {
        terminationStatus: process.terminationStatus)
    }

-    guard let arpCommandOutput = try pipe.fileHandleForReading.readToEnd() else {
-      throw ARPCommandYieldedInvalidOutputError(explanation: "empty output")
-    }
-
    self.arpCommandOutput = arpCommandOutput
  }

@@ -1,6 +1,6 @@
 import Foundation

 protocol Disk {
-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer]
-  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache?) async throws
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer]
+  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache?, deduplicate: Bool) async throws
 }
@@ -5,7 +5,7 @@ class DiskV1: Disk {
  private static let bufferSizeBytes = 4 * 1024 * 1024
  private static let layerLimitBytes = 500 * 1000 * 1000

-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer] {
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer] {
    var pushedLayers: [OCIManifestLayer] = []

    // Open the disk file
@@ -45,7 +45,7 @@ class DiskV1: Disk {
    return pushedLayers
  }

-  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache? = nil) async throws {
+  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache? = nil, deduplicate: Bool = false) async throws {
    if !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
      throw OCIError.FailedToCreateVmFile
    }
@@ -1,44 +1,89 @@
 import Foundation
 import Compression
+import System
+import Retry

 class DiskV2: Disk {
  private static let bufferSizeBytes = 4 * 1024 * 1024
-  private static let layerLimitBytes = 500 * 1000 * 1000
+  private static let layerLimitBytes = 512 * 1024 * 1024

-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer] {
-    var pushedLayers: [OCIManifestLayer] = []
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer] {
+    var pushedLayers: [(index: Int, pushedLayer: OCIManifestLayer)] = []

    // Open the disk file
    let mappedDisk = try Data(contentsOf: diskURL, options: [.alwaysMapped])

    // Compress the disk file as multiple individually decompressible streams,
    // each equal ``Self.layerLimitBytes`` bytes or less due to LZ4 compression
-    for data in mappedDisk.chunks(ofCount: layerLimitBytes) {
-      let compressedData = try (data as NSData).compressed(using: .lz4) as Data
+    try await withThrowingTaskGroup(of: (Int, OCIManifestLayer).self) { group in
+      for (index, data) in mappedDisk.chunks(ofCount: layerLimitBytes).enumerated() {
+        // Respect the concurrency limit
+        if index >= concurrency {
+          if let (index, pushedLayer) = try await group.next() {
+            pushedLayers.append((index, pushedLayer))
+          }
+        }

-      let layerDigest = try await registry.pushBlob(fromData: compressedData, chunkSizeMb: chunkSizeMb)
+        // Launch a disk layer pushing task
+        group.addTask {
+          let compressedData = try (data as NSData).compressed(using: .lz4) as Data
+          let compressedDataDigest = Digest.hash(compressedData)

-      pushedLayers.append(OCIManifestLayer(
-        mediaType: diskV2MediaType,
-        size: compressedData.count,
-        digest: layerDigest,
-        uncompressedSize: UInt64(data.count),
-        uncompressedContentDigest: Digest.hash(data)
-      ))
+          try await retry(maxAttempts: 5, backoff: .exponentialWithFullJitter(baseDelay: .seconds(5), maxDelay: .seconds(60))) {
+            if try await !registry.blobExists(compressedDataDigest) {
+              _ = try await registry.pushBlob(fromData: compressedData, chunkSizeMb: chunkSizeMb, digest: compressedDataDigest)
+            }
+          } recoverFromFailure: { error in
+            if error is URLError {
+              print("Error: \(error.localizedDescription)")
+              print("Attempting to re-try...")

-      // Update progress using a relative value
-      progress.completedUnitCount += Int64(data.count)
+              return .retry
+            }
+
+            return .throw
+          }
+
+          // Update progress using a relative value
+          progress.completedUnitCount += Int64(data.count)
+
+          return (index, OCIManifestLayer(
+            mediaType: diskV2MediaType,
+            size: compressedData.count,
+            digest: compressedDataDigest,
+            uncompressedSize: UInt64(data.count),
+            uncompressedContentDigest: Digest.hash(data)
+          ))
+        }
+      }
+
+      for try await pushedLayer in group {
+        pushedLayers.append(pushedLayer)
+      }
    }

-    return pushedLayers
+    return pushedLayers.sorted {
+      $0.index < $1.index
+    }.map {
+      $0.pushedLayer
+    }
  }

-  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache? = nil) async throws {
+  static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache? = nil, deduplicate: Bool = false) async throws {
    // Support resumable pulls
    let pullResumed = FileManager.default.fileExists(atPath: diskURL.path)

-    if !pullResumed && !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
-      throw OCIError.FailedToCreateVmFile
+    if !pullResumed {
+      if deduplicate, let localLayerCache = localLayerCache {
+        // Clone the local layer cache's disk and use it as a base, potentially
+        // reducing the space usage since some blocks won't be written at all
+        try FileManager.default.copyItem(at: localLayerCache.diskURL, to: diskURL)
+      } else {
+        // Otherwise create an empty disk
+        if !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
+          throw OCIError.FailedToCreateVmFile
+        }
+      }
    }

    // Calculate the uncompressed disk size
@@ -58,6 +103,15 @@ class DiskV2: Disk {
    try disk.truncate(atOffset: uncompressedDiskSize)
    try disk.close()

+    // Determine the file system block size
+    var st = stat()
+    if stat(diskURL.path, &st) == -1 {
+      let details = Errno(rawValue: errno)
+
+      throw RuntimeError.PullFailed("failed to stat(2) disk \(diskURL.path): \(details)")
+    }
+    let fsBlockSize = UInt64(st.st_blksize)
+
    // Concurrently fetch and decompress layers
    try await withThrowingTaskGroup(of: Void.self) { group in
      var globalDiskWritingOffset: UInt64 = 0
@@ -82,22 +136,46 @@ class DiskV2: Disk {
        // Launch a fetching and decompression task
        group.addTask {
          // No need to fetch and decompress anything if we've already done so
-          if try pullResumed && Digest.hash(diskURL, offset: diskWritingOffset, size: uncompressedLayerSize) == uncompressedLayerContentDigest {
-            // Update the progress
-            progress.completedUnitCount += Int64(diskLayer.size)
+          if pullResumed {
+            // do not check hash in the condition above to make it lazy e.g. only do expensive calculations if needed
+            if try Digest.hash(diskURL, offset: diskWritingOffset, size: uncompressedLayerSize) == uncompressedLayerContentDigest {
+              // Update the progress
+              progress.completedUnitCount += Int64(diskLayer.size)

-            return
+              return
+            }
          }

-          // Open the disk file
+          // Open the disk file for writing
          let disk = try FileHandle(forWritingTo: diskURL)

-          // Check if we already have this layer contents in the local layer cache
-          if let localLayerCache = localLayerCache, let data = localLayerCache.find(diskLayer.digest), Digest.hash(data) == uncompressedLayerContentDigest {
-            // Fulfil the layer contents from the local blob cache
-            _ = try zeroSkippingWrite(disk, diskWritingOffset, data)
+          // Also open the disk file for reading and verifying
+          // its contents in case the local layer cache is used
+          let rdisk: FileHandle? = if deduplicate && localLayerCache != nil {
+            try FileHandle(forReadingFrom: diskURL)
+          } else {
+            nil
+          }
+
+          // Check if we already have this layer contents in the local layer cache,
+          // or perhaps even on the cloned disk (when the deduplication is enabled)
+          if let localLayerCache = localLayerCache,
+             let localLayerInfo = localLayerCache.findInfo(digest: diskLayer.digest, offsetHint: diskWritingOffset),
+             localLayerInfo.uncompressedContentDigest == uncompressedLayerContentDigest {
+            if deduplicate && localLayerInfo.range.lowerBound == diskWritingOffset {
+              // Do nothing, because the data is already on the disk that we've inherited from
+            } else {
+              // Fulfil the layer contents from the local blob cache
+              let data = localLayerCache.subdata(localLayerInfo.range)
+              _ = try zeroSkippingWrite(disk, rdisk, fsBlockSize, diskWritingOffset, data)
+            }
+
            try disk.close()

+            if let rdisk = rdisk {
+              try rdisk.close()
+            }
+
            // Update the progress
            progress.completedUnitCount += Int64(diskLayer.size)

@@ -112,7 +190,7 @@ class DiskV2: Disk {
              return
            }

-            diskWritingOffset = try zeroSkippingWrite(disk, diskWritingOffset, data)
+            diskWritingOffset = try zeroSkippingWrite(disk, rdisk, fsBlockSize, diskWritingOffset, data)
          }

          try await registry.pullBlob(diskLayer.digest) { data in
@@ -125,6 +203,10 @@ class DiskV2: Disk {
          try filter.finalize()

          try disk.close()
+
+          if let rdisk = rdisk {
+            try rdisk.close()
+          }
        }

        globalDiskWritingOffset += uncompressedLayerSize
@@ -132,7 +214,7 @@ class DiskV2: Disk {
    }
  }

-  private static func zeroSkippingWrite(_ disk: FileHandle, _ offset: UInt64, _ data: Data) throws -> UInt64 {
+  private static func zeroSkippingWrite(_ disk: FileHandle, _ rdisk: FileHandle?, _ fsBlockSize: UInt64, _ offset: UInt64, _ data: Data) throws -> UInt64 {
    let holeGranularityBytes = 64 * 1024

    // A zero chunk for faster than byte-by-byte comparisons
@@ -152,7 +234,38 @@ class DiskV2: Disk {
    var offset = offset

    for chunk in data.chunks(ofCount: holeGranularityBytes) {
-      // Only write chunks that are not zero
+      // If the local layer cache is used, only write chunks that differ
+      // since the base disk can contain anything at any position
+      if let rdisk = rdisk {
+        // F_PUNCHHOLE requires the holes to be aligned to file system block boundaries
+        let isHoleAligned = (offset % fsBlockSize) == 0 && (UInt64(chunk.count) % fsBlockSize) == 0
+
+        if isHoleAligned && chunk == zeroChunk {
+          var arg = fpunchhole_t(fp_flags: 0, reserved: 0, fp_offset: off_t(offset), fp_length: off_t(chunk.count))
+
+          if fcntl(disk.fileDescriptor, F_PUNCHHOLE, &arg) == -1 {
+            let details = Errno(rawValue: errno)
+
+            throw RuntimeError.PullFailed("failed to punch hole: \(details)")
+          }
+        } else {
+          try rdisk.seek(toOffset: offset)
+          let actualContentsOnDisk = try rdisk.read(upToCount: chunk.count)
+
+          if chunk != actualContentsOnDisk {
+            try disk.seek(toOffset: offset)
+            disk.write(chunk)
+          }
+        }
+
+        offset += UInt64(chunk.count)
+
+        continue
+      }
+
+      // Otherwise, only write chunks that are not zero
+      // since the base disk is created from scratch and
+      // is zeroed via truncate(2)
      if chunk != zeroChunk {
        try disk.seek(toOffset: offset)
        disk.write(chunk)
@@ -78,7 +78,7 @@ struct OCIManifestConfig: Codable, Equatable {
  var digest: String
 }

-struct OCIManifestLayer: Codable, Equatable {
+struct OCIManifestLayer: Codable, Equatable, Hashable {
  var mediaType: String
  var size: Int
  var digest: String
@@ -113,6 +113,14 @@ struct OCIManifestLayer: Codable, Equatable {
  func uncompressedContentDigest() -> String? {
    annotations?[uncompressedContentDigestAnnotation]
  }
+
+  static func == (lhs: Self, rhs: Self) -> Bool {
+    return lhs.digest == rhs.digest
+  }
+
+  func hash(into hasher: inout Hasher) {
+    hasher.combine(digest)
+  }
 }

 struct Descriptor: Equatable {
@@ -1,4 +1,4 @@
-// Generated from java-escape by ANTLR 4.11.1
+// Generated from Reference.g4 by ANTLR 4.13.2

 import Antlr4

@@ -1,4 +1,4 @@
-// Generated from java-escape by ANTLR 4.11.1
+// Generated from Reference.g4 by ANTLR 4.13.2
 import Antlr4

 open class ReferenceLexer: Lexer {
@@ -49,7 +49,7 @@ open class ReferenceLexer: Lexer {

 	public
 	required init(_ input: CharStream) {
-	    RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION)
+	    RuntimeMetaData.checkVersion("4.13.2", RuntimeMetaData.VERSION)
 		super.init(input)
 		_interp = LexerATNSimulator(self, ReferenceLexer._ATN, ReferenceLexer._decisionToDFA, ReferenceLexer._sharedContextCache)
 	}
@@ -1,4 +1,4 @@
-// Generated from java-escape by ANTLR 4.11.1
+// Generated from Reference.g4 by ANTLR 4.13.2
 import Antlr4

 /**
@@ -1,4 +1,4 @@
-// Generated from java-escape by ANTLR 4.11.1
+// Generated from Reference.g4 by ANTLR 4.13.2
 import Antlr4

 open class ReferenceParser: Parser {
@@ -41,7 +41,7 @@ open class ReferenceParser: Parser {
 	static let VOCABULARY = Vocabulary(_LITERAL_NAMES, _SYMBOLIC_NAMES)

 	override open
-	func getGrammarFileName() -> String { return "java-escape" }
+	func getGrammarFileName() -> String { return "Reference.g4" }

 	override open
 	func getRuleNames() -> [String] { return ReferenceParser.ruleNames }
@@ -60,7 +60,7 @@ open class ReferenceParser: Parser {

 	override public
 	init(_ input:TokenStream) throws {
-	    RuntimeMetaData.checkVersion("4.11.1", RuntimeMetaData.VERSION)
+	    RuntimeMetaData.checkVersion("4.13.2", RuntimeMetaData.VERSION)
 		try super.init(input)
 		_interp = ParserATNSimulator(self,ReferenceParser._ATN,ReferenceParser._decisionToDFA, ReferenceParser._sharedContextCache)
 	}
@@ -460,7 +460,7 @@ open class ReferenceParser: Parser {
 		 		setState(63)
 		 		try _errHandler.sync(self)
 		 		_la = try _input.LA(1)
-		 		if ((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0) {
+		 		if (((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0)) {
 		 			setState(62)
 		 			try separator()

@@ -611,7 +611,7 @@ open class ReferenceParser: Parser {
 		 	setState(84)
 		 	try _errHandler.sync(self)
 		 	_la = try _input.LA(1)
-		 	while ((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0) {
+		 	while (((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0)) {
 		 		setState(79)
 		 		try separator()
 		 		setState(80)
@@ -664,7 +664,7 @@ open class ReferenceParser: Parser {
 		 	try enterOuterAlt(_localctx, 1)
 		 	setState(87)
 		 	_la = try _input.LA(1)
-		 	if (!((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0)) {
+		 	if (!(((Int64(_la) & ~0x3f) == 0 && ((Int64(1) << _la) & 88) != 0))) {
 		 	try _errHandler.recoverInline(self)
 		 	}
 		 	else {
@@ -10,6 +10,7 @@ enum RegistryError: Error {
 }

 enum HTTPMethod: String {
+  case HEAD = "HEAD"
  case GET = "GET"
  case POST = "POST"
  case PUT = "PUT"
@@ -21,6 +22,7 @@ enum HTTPCode: Int {
  case Created = 201
  case Accepted = 202
  case Unauthorized = 401
+  case NotFound = 404
 }

 extension Data {
@@ -112,11 +114,11 @@ class Registry {
    return host
  }

-  init(urlComponents: URLComponents,
+  init(baseURL: URL,
       namespace: String,
       credentialsProviders: [CredentialsProvider] = [EnvironmentCredentialsProvider(), DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
  ) throws {
-    baseURL = urlComponents.url!
+    self.baseURL = baseURL
    self.namespace = namespace
    self.credentialsProviders = credentialsProviders
  }
@@ -130,7 +132,17 @@ class Registry {
    let proto = insecure ? "http" : "https"
    let baseURLComponents = URLComponents(string: proto + "://" + host + "/v2/")!

-    try self.init(urlComponents: baseURLComponents, namespace: namespace, credentialsProviders: credentialsProviders)
+    guard let baseURL = baseURLComponents.url else {
+      var hint = ""
+
+      if host.hasPrefix("http://") || host.hasPrefix("https://") {
+        hint += ", make sure that it doesn't start with http:// or https://"
+      }
+
+      throw RuntimeError.ImproperlyFormattedHost(host, hint)
+    }
+
+    try self.init(baseURL: baseURL, namespace: namespace, credentialsProviders: credentialsProviders)
  }

  func ping() async throws {
@@ -179,7 +191,7 @@ class Registry {
    return URLComponents(url: uploadLocation.absolutize(baseURL), resolvingAgainstBaseURL: true)!
  }

-  public func pushBlob(fromData: Data, chunkSizeMb: Int = 0) async throws -> String {
+  public func pushBlob(fromData: Data, chunkSizeMb: Int = 0, digest: String? = nil) async throws -> String {
    // Initiate a blob upload
    let (data, postResponse) = try await dataRequest(.POST, endpointURL("\(namespace)/blobs/uploads/"),
                                                     headers: ["Content-Length": "0"])
@@ -191,7 +203,7 @@ class Registry {
    // Figure out where to upload the blob
    var uploadLocation = try uploadLocationFromResponse(postResponse)

-    let digest = Digest.hash(fromData)
+    let digest = digest ?? Digest.hash(fromData)

    if chunkSizeMb == 0 {
      // monolithic upload
@@ -239,6 +251,19 @@ class Registry {
    return digest
  }

+  public func blobExists(_ digest: String) async throws -> Bool {
+    let (data, response) = try await dataRequest(.HEAD, endpointURL("\(namespace)/blobs/\(digest)"))
+
+    switch response.statusCode {
+    case HTTPCode.Ok.rawValue:
+      return true
+    case HTTPCode.NotFound.rawValue:
+      return false
+    default:
+      throw RegistryError.UnexpectedHTTPStatusCode(when: "checking blob", code: response.statusCode, details: data.asText())
+    }
+  }
+
  public func pullBlob(_ digest: String, handler: (Data) async throws -> Void) async throws {
    let (channel, response) = try await channelRequest(.GET, endpointURL("\(namespace)/blobs/\(digest)"), viaFile: true)
    if response.statusCode != HTTPCode.Ok.rawValue {
@@ -302,12 +327,12 @@ class Registry {
      request.httpBody = body
    }

-    var (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile)
+    var (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile, doAuth: doAuth)

    if doAuth && response.statusCode == HTTPCode.Unauthorized.rawValue {
      _ = try await channel.asData()
      try await auth(response: response)
-      (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile)
+      (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile, doAuth: doAuth)
    }

    return (channel, response)
@@ -388,11 +413,13 @@ class Registry {
    return nil
  }

-  private func authAwareRequest(request: URLRequest, viaFile: Bool = false) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
+  private func authAwareRequest(request: URLRequest, viaFile: Bool = false, doAuth: Bool) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
    var request = request

-    if let (name, value) = await authenticationKeeper.header() {
-      request.addValue(value, forHTTPHeaderField: name)
+    if doAuth {
+      if let (name, value) = await authenticationKeeper.header() {
+        request.addValue(value, forHTTPHeaderField: name)
+      }
    }

    request.setValue("Tart/\(CI.version) (\(DeviceInfo.os); \(DeviceInfo.model))",
@@ -11,7 +11,7 @@ class PIDLock {
    if fd == -1 {
      let details = Errno(rawValue: CInt(errno))

-      throw RuntimeError.PIDLockFailed("failed to open lock file \(url): \(details)")
+      throw RuntimeError.PIDLockMissing("failed to open lock file \(url): \(details)")
    }
  }

@@ -8,7 +8,7 @@ struct UnsupportedHostOSError: Error, CustomStringConvertible {

 #if arch(arm64)

-  struct Darwin: Platform {
+  struct Darwin: PlatformSuspendable {
    var ecid: VZMacMachineIdentifier
    var hardwareModel: VZMacHardwareModel

@@ -58,7 +58,11 @@ struct UnsupportedHostOSError: Error, CustomStringConvertible {
      VZMacOSBootLoader()
    }

-    func platform(nvramURL: URL) throws -> VZPlatformConfiguration {
+    func platform(nvramURL: URL, needsNestedVirtualization: Bool) throws -> VZPlatformConfiguration {
+      if needsNestedVirtualization {
+        throw RuntimeError.VMConfigurationError("macOS virtual machines do not support nested virtualization")
+      }
+
      let result = VZMacPlatformConfiguration()

      result.machineIdentifier = ecid
@@ -103,18 +107,32 @@ struct UnsupportedHostOSError: Error, CustomStringConvertible {
    func keyboards() -> [VZKeyboardConfiguration] {
      if #available(macOS 14, *) {
        // Mac keyboard is only supported by guests starting with macOS Ventura
-        return [VZMacKeyboardConfiguration()]
+        return [VZUSBKeyboardConfiguration(), VZMacKeyboardConfiguration()]
      } else {
        return [VZUSBKeyboardConfiguration()]
      }
    }

+    func keyboardsSuspendable() -> [VZKeyboardConfiguration] {
+      if #available(macOS 14, *) {
+        return [VZMacKeyboardConfiguration()]
+      } else {
+        // fallback to the regular configuration
+        return keyboards()
+      }
+    }
+
    func pointingDevices() -> [VZPointingDeviceConfiguration] {
-      if #available(macOS 13, *) {
+      // Trackpad is only supported by guests starting with macOS Ventura
+      [VZUSBScreenCoordinatePointingDeviceConfiguration(), VZMacTrackpadConfiguration()]
+    }
+
+    func pointingDevicesSuspendable() -> [VZPointingDeviceConfiguration] {
+      if #available(macOS 14, *) {
        return [VZMacTrackpadConfiguration()]
      } else {
        // fallback to the regular configuration
-        return [VZUSBScreenCoordinatePointingDeviceConfiguration()]
+        return pointingDevices()
      }
    }
  }
@@ -14,8 +14,12 @@ struct Linux: Platform {
    return result
  }

-  func platform(nvramURL: URL) throws -> VZPlatformConfiguration {
-    VZGenericPlatformConfiguration()
+  func platform(nvramURL: URL, needsNestedVirtualization: Bool) throws -> VZPlatformConfiguration {
+    let config = VZGenericPlatformConfiguration()
+    if #available(macOS 15, *) {
+      config.isNestedVirtualizationEnabled = needsNestedVirtualization
+    }
+    return config
  }

  func graphicsDevice(vmConfig: VMConfig) -> VZGraphicsDeviceConfiguration {
@@ -3,8 +3,13 @@ import Virtualization
 protocol Platform: Codable {
  func os() -> OS
  func bootLoader(nvramURL: URL) throws -> VZBootLoader
-  func platform(nvramURL: URL) throws -> VZPlatformConfiguration
+  func platform(nvramURL: URL, needsNestedVirtualization: Bool) throws -> VZPlatformConfiguration
  func graphicsDevice(vmConfig: VMConfig) -> VZGraphicsDeviceConfiguration
  func keyboards() -> [VZKeyboardConfiguration]
  func pointingDevices() -> [VZPointingDeviceConfiguration]
 }
+
+protocol PlatformSuspendable: Platform {
+  func pointingDevicesSuspendable() -> [VZPointingDeviceConfiguration]
+  func keyboardsSuspendable() -> [VZKeyboardConfiguration]
+}
@@ -1,4 +1,5 @@
 import Foundation
+import System

 extension URL {
  func accessDate() throws -> Date {
@@ -13,7 +14,9 @@ extension URL {
    let times = [accessDate.asTimeval(), modificationDate.asTimeval()]
    let ret = utimes(path, times)
    if ret != 0 {
-      throw RuntimeError.FailedToUpdateAccessDate("utimes(2) failed: \(ret.explanation())")
+      let details = Errno(rawValue: CInt(errno))
+
+      throw RuntimeError.FailedToUpdateAccessDate("utimes(2) failed: \(details)")
    }
  }
 }
@@ -1,4 +1,5 @@
 import Foundation
+import XAttr

 extension URL: Prunable {
  var url: URL {
@@ -13,7 +14,31 @@ extension URL: Prunable {
    try resourceValues(forKeys: [.totalFileAllocatedSizeKey]).totalFileAllocatedSize!
  }

+  func deduplicatedSizeBytes() throws -> Int {
+    let values = try resourceValues(forKeys: [.totalFileAllocatedSizeKey, .mayShareFileContentKey])
+    // make sure the file's origin file is there and duplication works
+    if values.mayShareFileContent == true {
+      return Int(deduplicatedBytes())
+    }
+    return 0
+  }
+
  func sizeBytes() throws -> Int {
    try resourceValues(forKeys: [.totalFileSizeKey]).totalFileSize!
  }
+
+  func setDeduplicatedBytes(_ size: UInt64) {
+    let data = "\(size)".data(using: .utf8)!
+    try! self.setExtendedAttribute(name: "run.tart.deduplicated-bytes", value: data)
+  }
+
+  func deduplicatedBytes() -> UInt64 {
+    guard let data = try? self.extendedAttributeValue(forName: "run.tart.deduplicated-bytes") else {
+      return 0
+    }
+    if let strValue = String(data: data, encoding: .utf8) {
+      return UInt64(strValue) ?? 0
+    }
+    return 0
+  }
 }
@@ -47,8 +47,10 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
       directorySharingDevices: [VZDirectorySharingDeviceConfiguration] = [],
       serialPorts: [VZSerialPortConfiguration] = [],
       suspendable: Bool = false,
+       nested: Bool = false,
       audio: Bool = true,
-       clipboard: Bool = true
+       clipboard: Bool = true,
+       sync: VZDiskImageSynchronizationMode = .full
  ) throws {
    name = vmDir.name
    config = try VMConfig.init(fromURL: vmDir.configURL)
@@ -65,8 +67,10 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
                                                directorySharingDevices: directorySharingDevices,
                                                serialPorts: serialPorts,
                                                suspendable: suspendable,
+                                                nested: nested,
                                                audio: audio,
-                                                clipboard: clipboard
+                                                clipboard: clipboard,
+                                                sync: sync
    )
    virtualMachine = VZVirtualMachine(configuration: configuration)

@@ -293,8 +297,10 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
    directorySharingDevices: [VZDirectorySharingDeviceConfiguration],
    serialPorts: [VZSerialPortConfiguration],
    suspendable: Bool = false,
+    nested: Bool = false,
    audio: Bool = true,
-    clipboard: Bool = true
+    clipboard: Bool = true,
+    sync: VZDiskImageSynchronizationMode = .full
  ) throws -> VZVirtualMachineConfiguration {
    let configuration = VZVirtualMachineConfiguration()

@@ -306,25 +312,37 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
    configuration.memorySize = vmConfig.memorySize

    // Platform
-    configuration.platform = try vmConfig.platform.platform(nvramURL: nvramURL)
+    configuration.platform = try vmConfig.platform.platform(nvramURL: nvramURL, needsNestedVirtualization: nested)

    // Display
    configuration.graphicsDevices = [vmConfig.platform.graphicsDevice(vmConfig: vmConfig)]

    // Audio
+    let soundDeviceConfiguration = VZVirtioSoundDeviceConfiguration()
+
    if audio && !suspendable {
-      let soundDeviceConfiguration = VZVirtioSoundDeviceConfiguration()
      let inputAudioStreamConfiguration = VZVirtioSoundDeviceInputStreamConfiguration()
-      inputAudioStreamConfiguration.source = VZHostAudioInputStreamSource()
      let outputAudioStreamConfiguration = VZVirtioSoundDeviceOutputStreamConfiguration()
+
+      inputAudioStreamConfiguration.source = VZHostAudioInputStreamSource()
      outputAudioStreamConfiguration.sink = VZHostAudioOutputStreamSink()
+
      soundDeviceConfiguration.streams = [inputAudioStreamConfiguration, outputAudioStreamConfiguration]
-      configuration.audioDevices = [soundDeviceConfiguration]
+    } else {
+      // just a null speaker
+      soundDeviceConfiguration.streams = [VZVirtioSoundDeviceOutputStreamConfiguration()]
    }

+    configuration.audioDevices = [soundDeviceConfiguration]
+
    // Keyboard and mouse
-    configuration.keyboards = vmConfig.platform.keyboards()
-    configuration.pointingDevices = vmConfig.platform.pointingDevices()
+    if suspendable, let platformSuspendable = vmConfig.platform.self as? PlatformSuspendable {
+      configuration.keyboards = platformSuspendable.keyboardsSuspendable()
+      configuration.pointingDevices = platformSuspendable.pointingDevicesSuspendable()
+    } else {
+      configuration.keyboards = vmConfig.platform.keyboards()
+      configuration.pointingDevices = vmConfig.platform.pointingDevices()
+    }

    // Networking
    configuration.networkDevices = network.attachments().map {
@@ -347,16 +365,10 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
    // Storage
    let attachment: VZDiskImageStorageDeviceAttachment = vmConfig.os == .linux ?
      // Use "cached" caching mode for virtio drive to prevent fs corruption on linux
-      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .cached, synchronizationMode: .full) :
-      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false)
+      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .cached, synchronizationMode: sync) :
+      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .automatic, synchronizationMode: sync)

-    var device: VZStorageDeviceConfiguration
-    if #available(macOS 14, *), vmConfig.os == .linux {
-      device = VZNVMExpressControllerDeviceConfiguration(attachment: attachment)
-    } else {
-      device = VZVirtioBlockDeviceConfiguration(attachment: attachment)
-    }
-    var devices: [VZStorageDeviceConfiguration] = [device]
+    var devices: [VZStorageDeviceConfiguration] = [VZVirtioBlockDeviceConfiguration(attachment: attachment)]
    devices.append(contentsOf: additionalStorageDevices)
    configuration.storageDevices = devices

@@ -11,10 +11,7 @@ enum OCIError: Error {
 }

 extension VMDirectory {
-  private static let bufferSizeBytes = 64 * 1024 * 1024
-  private static let layerLimitBytes = 500 * 1000 * 1000
-
-  func pullFromRegistry(registry: Registry, manifest: OCIManifest, concurrency: UInt, localLayerCache: LocalLayerCache?) async throws {
+  func pullFromRegistry(registry: Registry, manifest: OCIManifest, concurrency: UInt, localLayerCache: LocalLayerCache?, deduplicate: Bool) async throws {
    // Pull VM's config file layer and re-serialize it into a config file
    let configLayers = manifest.layers.filter {
      $0.mediaType == configMediaType
@@ -57,11 +54,17 @@ extension VMDirectory {
    do {
      try await diskImplType.pull(registry: registry, diskLayers: layers, diskURL: diskURL,
                                  concurrency: concurrency, progress: progress,
-                                  localLayerCache: localLayerCache)
+                                  localLayerCache: localLayerCache,
+                                  deduplicate: deduplicate)
    } catch let error where error is FilterError {
      throw RuntimeError.PullFailed("failed to decompress disk: \(error.localizedDescription)")
    }

+    if deduplicate, let llc = localLayerCache {
+      // set custom attribute to remember deduplicated bytes
+      diskURL.setDeduplicatedBytes(llc.deduplicatedBytes)
+    }
+
    // Pull VM's NVRAM file layer and store it in an NVRAM file
    defaultLogger.appendNewLine("pulling NVRAM...")

@@ -79,9 +82,12 @@ extension VMDirectory {
      nvram.write(data)
    }
    try nvram.close()
+
+    // Serialize VM's manifest to enable better deduplication on subsequent "tart pull"'s
+    try manifest.toJSON().write(to: manifestURL)
  }

-  func pushToRegistry(registry: Registry, references: [String], chunkSizeMb: Int, diskFormat: String) async throws -> RemoteName {
+  func pushToRegistry(registry: Registry, references: [String], chunkSizeMb: Int, diskFormat: String, concurrency: UInt) async throws -> RemoteName {
    var layers = Array<OCIManifestLayer>()

    // Read VM's config and push it as blob
@@ -100,9 +106,9 @@ extension VMDirectory {

    switch diskFormat {
    case "v1":
-      layers.append(contentsOf: try await DiskV1.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, progress: progress))
+      layers.append(contentsOf: try await DiskV1.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, concurrency: concurrency, progress: progress))
    case "v2":
-      layers.append(contentsOf: try await DiskV2.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, progress: progress))
+      layers.append(contentsOf: try await DiskV2.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, concurrency: concurrency, progress: progress))
    default:
      throw RuntimeError.OCIUnsupportedDiskFormat(diskFormat)
    }
@@ -23,6 +23,9 @@ struct VMDirectory: Prunable {
  var stateURL: URL {
    baseURL.appendingPathComponent("state.vzvmsave")
  }
+  var manifestURL: URL {
+    baseURL.appendingPathComponent("manifest.json")
+  }

  var explicitlyPulledMark: URL {
    baseURL.appendingPathComponent(".explicitly-pulled")
@@ -179,6 +182,14 @@ struct VMDirectory: Prunable {
    try allocatedSizeBytes() / 1000 / 1000 / 1000
  }

+  func deduplicatedSizeBytes() throws -> Int {
+    try configURL.deduplicatedSizeBytes() + diskURL.deduplicatedSizeBytes() + nvramURL.deduplicatedSizeBytes()
+  }
+
+  func deduplicatedSizeGB() throws -> Int {
+    try deduplicatedSizeBytes() / 1000 / 1000 / 1000
+  }
+
  func sizeBytes() throws -> Int {
    try configURL.sizeBytes() + diskURL.sizeBytes() + nvramURL.sizeBytes()
  }
@@ -24,6 +24,8 @@ class VMStorageHelper {
  private static func missingVMWrap<R: Any>(_ name: String, closure: () throws -> R) throws -> R {
    do {
      return try closure()
+    } catch RuntimeError.PIDLockMissing {
+      throw RuntimeError.VMDoesNotExist(name: name)
    } catch {
      if error.isFileNotFound() {
        throw RuntimeError.VMDoesNotExist(name: name)
@@ -59,8 +61,10 @@ enum RuntimeError : Error {
  case InvalidDiskSize(_ message: String)
  case FailedToUpdateAccessDate(_ message: String)
  case PIDLockFailed(_ message: String)
+  case PIDLockMissing(_ message: String)
  case FailedToParseRemoteName(_ message: String)
  case VMTerminationFailed(_ message: String)
+  case ImproperlyFormattedHost(_ host: String, _ hint: String)
  case InvalidCredentials(_ message: String)
  case VMDirectoryAlreadyInitialized(_ message: String)
  case ExportFailed(_ message: String)
@@ -70,6 +74,7 @@ enum RuntimeError : Error {
  case OCIUnsupportedDiskFormat(_ format: String)
  case SuspendFailed(_ message: String)
  case PullFailed(_ message: String)
+  case VirtualMachineLimitExceeded(_ hint: String)
 }

 protocol HasExitCode {
@@ -103,10 +108,14 @@ extension RuntimeError : CustomStringConvertible {
      return message
    case .PIDLockFailed(let message):
      return message
+    case .PIDLockMissing(let message):
+      return message
    case .FailedToParseRemoteName(let cause):
      return "failed to parse remote name: \(cause)"
    case .VMTerminationFailed(let message):
      return message
+    case .ImproperlyFormattedHost(let host, let hint):
+      return "improperly formatted host \"\(host)\" was provided\(hint)"
    case .InvalidCredentials(let message):
      return message
    case .VMDirectoryAlreadyInitialized(let message):
@@ -125,6 +134,8 @@ extension RuntimeError : CustomStringConvertible {
      return "Failed to suspend the VM: \(message)"
    case .PullFailed(let message):
      return message
+    case .VirtualMachineLimitExceeded(let hint):
+      return "The number of VMs exceeds the system limit\(hint)"
    }
  }
 }
@@ -66,7 +66,7 @@ class VMStorageLocal: PrunableStorage {
  }

  func prunables() throws -> [Prunable] {
-    try list().map { (_, vmDir) in vmDir }
+    try list().map { (_, vmDir) in vmDir }.filter { try !$0.running() }
  }

  func hasVMsWithMACAddress(macAddress: String) throws -> Bool {
@@ -140,7 +140,7 @@ class VMStorageOCI: PrunableStorage {
    try list().filter { (_, _, isSymlink) in !isSymlink }.map { (_, vmDir, _) in vmDir }
  }

-  func pull(_ name: RemoteName, registry: Registry, concurrency: UInt) async throws {
+  func pull(_ name: RemoteName, registry: Registry, concurrency: UInt, deduplicate: Bool) async throws {
    SentrySDK.configureScope { scope in
      scope.setContext(value: ["imageName": name.description], key: "OCI")
    }
@@ -197,21 +197,29 @@ class VMStorageOCI: PrunableStorage {

      try await withTaskCancellationHandler(operation: {
        try await retry(maxAttempts: 5, backoff: .exponentialWithFullJitter(baseDelay: .seconds(5), maxDelay: .seconds(60))) {
-          var localLayerCache: LocalLayerCache? = nil
+          // Choose the best base image which has the most deduplication ratio
+          let localLayerCache = try await chooseLocalLayerCache(name, manifest, registry)

-          if name.reference.type == .Tag,
-             let vmDir = try? open(name),
-             let digest = try? digest(name),
-             let (manifest, _) = try? await registry.pullManifest(reference: digest) {
-            localLayerCache = try LocalLayerCache(vmDir.diskURL, manifest)
+          if let llc = localLayerCache {
+            let deduplicatedHuman = ByteCountFormatter.string(fromByteCount: Int64(llc.deduplicatedBytes), countStyle: .file)
+
+            if deduplicate {
+              defaultLogger.appendNewLine("found an image \(llc.name) that will allow us to deduplicate \(deduplicatedHuman), using it as a base...")
+            } else {
+              defaultLogger.appendNewLine("found an image \(llc.name) that will allow us to avoid fetching \(deduplicatedHuman), will try use it...")
+            }
          }

-          try await tmpVMDir.pullFromRegistry(registry: registry, manifest: manifest, concurrency: concurrency, localLayerCache: localLayerCache)
+          try await tmpVMDir.pullFromRegistry(registry: registry, manifest: manifest, concurrency: concurrency, localLayerCache: localLayerCache, deduplicate: deduplicate)
        } recoverFromFailure: { error in
-          print("Error: \(error.localizedDescription)")
-          print("Attempting to re-try...")
+          if error is URLError {
+            print("Error: \(error.localizedDescription)")
+            print("Attempting to re-try...")

-          return .retry
+            return .retry
+          }
+
+          return .throw
        }
        try move(digestName, from: tmpVMDir)
        transaction.finish()
@@ -231,6 +239,9 @@ class VMStorageOCI: PrunableStorage {
      // are excluded from garbage collection
      VMDirectory(baseURL: vmURL(name)).markExplicitlyPulled()
    }
+
+    // to explicitly set the image as being accessed so it won't get pruned immediately
+    _ = try VMStorageOCI().open(name)
  }

  func linked(from: RemoteName, to: RemoteName) -> Bool {
@@ -249,6 +260,59 @@ class VMStorageOCI: PrunableStorage {

    try gc()
  }
+
+  func chooseLocalLayerCache(_ name: RemoteName, _ manifest: OCIManifest, _ registry: Registry) async throws -> LocalLayerCache? {
+    // Establish a closure that will calculate how much bytes
+    // we'll deduplicate if we re-use the given manifest
+    let target = Swift.Set(manifest.layers)
+
+    let calculateDeduplicatedBytes = { (manifest: OCIManifest) -> UInt64 in
+      target.intersection(manifest.layers).map({ UInt64($0.size) }).reduce(0, +)
+    }
+
+    // Load OCI VM images and their manifests (if present)
+    var candidates: [(name: String, vmDir: VMDirectory, manifest: OCIManifest, deduplicatedBytes: UInt64)] = []
+
+    for (name, vmDir, isSymlink) in try list() {
+      if isSymlink {
+        continue
+      }
+
+      guard let manifestJSON = try? Data(contentsOf: vmDir.manifestURL) else {
+        continue
+      }
+
+      guard let manifest = try? OCIManifest(fromJSON: manifestJSON) else {
+        continue
+      }
+
+      candidates.append((name, vmDir, manifest, calculateDeduplicatedBytes(manifest)))
+    }
+
+    // Previously we haven't stored the OCI VM image manifests, but still fetched the VM image manifest if
+    // what the user was trying to pull was a tagged image, and we already had that image in the OCI VM cache
+    //
+    // Keep supporting this behavior for backwards comaptibility, but only communicate
+    // with the registry if we haven't already retrieved the manifest for that OCI VM image.
+    if name.reference.type == .Tag,
+       let vmDir = try? open(name),
+       let digest = try? digest(name),
+       try !candidates.contains(where: {try $0.manifest.digest() == digest}),
+       let (manifest, _) = try? await registry.pullManifest(reference: digest) {
+      candidates.append((name.description, vmDir, manifest, calculateDeduplicatedBytes(manifest)))
+    }
+
+    // Now, find the best match based on how many bytes we'll deduplicate
+    let choosen = candidates.filter {
+      $0.deduplicatedBytes > 1024 * 1024 * 1024 // save at least 1GB
+    }.max { left, right in
+      return left.deduplicatedBytes < right.deduplicatedBytes
+    }
+
+    return try choosen.flatMap({ choosen in
+      try LocalLayerCache(choosen.name, choosen.deduplicatedBytes, choosen.vmDir.diskURL, choosen.manifest)
+    })
+  }
 }

 extension URL {
@@ -36,7 +36,7 @@ final class LayerizerTests: XCTestCase {
    let pulledDiskFileURL = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)

    print("pushing disk...")
-    let diskLayers = try await DiskV1.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, progress: Progress())
+    let diskLayers = try await DiskV1.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, concurrency: 4, progress: Progress())

    print("pulling disk...")
    try await DiskV1.pull(registry: registry, diskLayers: diskLayers, diskURL: pulledDiskFileURL, concurrency: 16, progress: Progress())
@@ -57,7 +57,7 @@ final class LayerizerTests: XCTestCase {
    let pulledDiskFileURL = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)

    print("pushing disk...")
-    let diskLayers = try await DiskV2.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, progress: Progress())
+    let diskLayers = try await DiskV2.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, concurrency: 4, progress: Progress())

    print("pulling disk...")
    try await DiskV2.pull(registry: registry, diskLayers: diskLayers, diskURL: pulledDiskFileURL, concurrency: 16, progress: Progress())
@@ -39,7 +39,7 @@ class RegistryRunner {
    let port = try Self.dockerCmd("inspect", containerID, "--format", "{{(index (index .NetworkSettings.Ports \"5000/tcp\") 0).HostPort}}")
      .trimmingCharacters(in: CharacterSet.newlines)

-    registry = try Registry(urlComponents: URLComponents(string: "http://127.0.0.1:\(port)/v2/")!,
+    registry = try Registry(baseURL: URL(string: "http://127.0.0.1:\(port)/v2/")!,
                            namespace: "vm-image")

    // Wait for the Docker Registry to start
@@ -5,7 +5,7 @@ Tart comes with a Golang-based benchmarking utility that allows one to easily co
 Currently, only Flexible I/O tester workloads are supported. To run them, [install Golang](https://go.dev/) and run the following command from this (`benchmark/`) directory:

 ```shell
-go run cmd/main.go fio
+go run cmd/main.go fio --image ghcr.io/cirruslabs/macos-sonoma-base:latest
 ```

 You can also enable the debugging output to diagnose issues:
@@ -15,6 +15,7 @@ import (
 )

 var debug bool
+var image string

 func NewCommand() *cobra.Command {
 	cmd := &cobra.Command{
@@ -24,6 +25,7 @@ func NewCommand() *cobra.Command {
 	}

 	cmd.Flags().BoolVar(&debug, "debug", false, "enable debug logging")
+	cmd.Flags().StringVar(&image, "image", "ghcr.io/cirruslabs/macos-sonoma-base:latest", "image to use for testing")

 	return cmd
 }
@@ -111,7 +113,7 @@ func initializeExecutors(ctx context.Context, logger *zap.Logger) ([]executor.Ex

 	logger.Info("initializing Tart executor")

-	tart, err := tart.New(ctx, logger)
+	tart, err := tart.New(ctx, image, logger)
 	if err != nil {
 		return nil, err
 	}
@@ -14,8 +14,6 @@ import (
 	"time"
 )

-const baseImage = "ghcr.io/cirruslabs/macos-sonoma-base:latest"
-
 type Tart struct {
 	vmRunCancel context.CancelFunc
 	vmName      string
@@ -23,17 +21,17 @@ type Tart struct {
 	logger      *zap.Logger
 }

-func New(ctx context.Context, logger *zap.Logger) (*Tart, error) {
+func New(ctx context.Context, image string, logger *zap.Logger) (*Tart, error) {
 	tart := &Tart{
 		vmName: fmt.Sprintf("tart-benchmark-%s", uuid.NewString()),
 		logger: logger,
 	}

-	if err := Cmd(ctx, tart.logger, "pull", baseImage); err != nil {
+	if err := Cmd(ctx, tart.logger, "pull", image); err != nil {
 		return nil, err
 	}

-	if err := Cmd(ctx, tart.logger, "clone", baseImage, tart.vmName); err != nil {
+	if err := Cmd(ctx, tart.logger, "clone", image, tart.vmName); err != nil {
 		return nil, err
 	}

@@ -11,7 +11,7 @@ import (
 func TestTart(t *testing.T) {
 	ctx := context.Background()

-	tart, err := tart.New(ctx, zap.NewNop())
+	tart, err := tart.New(ctx, "ghcr.io/cirruslabs/macos-sonoma-base:latest", zap.NewNop())
 	require.NoError(t, err)

 	output, err := tart.Run(ctx, "echo \"this is a test\"")
@@ -90,3 +90,21 @@ Instead of Anka Registry, Tart can work with any OCI-compatible container regist
 and scalable experience for distributing virtual machines.

 Tart doesn't yet have an analogue of Anka Controller for managing long living VMs but [soon will be](https://github.com/cirruslabs/tart/issues/372).
+
+## Automatic pruning
+
+`tart pull` and `tart clone` commands check the remaining space available on the volume associated with `TART_HOME` directory (defaults to `~/.tart`) before pulling or cloning anything.
+
+In case there's not enough space to fit the newly pulled or cloned VM image, Tart will remove the least recently accessed VMs from OCI cache and `.ipsw` files from IPSW cache until enough free space is available.
+
+To disable this functionality, set the `TART_NO_AUTO_PRUNE` environment variable either globally:
+
+```shell
+export TART_NO_AUTO_PRUNE=
+```
+
+...or per `tart pull` and `tart clone` invocation as follows:
+
+```shell
+TART_NO_AUTO_PRUNE= tart pull ...
+```
@@ -0,0 +1,65 @@
+## Architecture
+
+Orchard cluster consists of two components:
+
+* Controller — responsible for managing the cluster and scheduling of resources
+* Worker — responsible for executing the VMs
+* Client — responsible for creating, modifying and removing the resources on the Controller, can either be an Orchard CLI or [an API consumer](/orchard/integration-guide)
+
+Normally you deploy a single Controller that needs to be accessible to both the Clients and Workers. Then you can deploy the Workers, which can reside anywhere and be inaccessible to Clients directly, e.g. behind a NAT.
+
+## Security
+
+When an Orchard Client or a Worker connects to the Controller, they need to establish trust and verify that they're talking to the right Controller, so that no [man-in-the-middle attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) is possible.
+
+Similarly to web-browsers (that rely on the [public key infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure)) and SSH (which relies on semi-automated fingerprint verification), Orchard combines these two traits in a hybrid approach by defaulting to automatic PKI verification (can be disabled by [`--no-pki`](#--no-pki-override)) and falling-back to a manual verification for self-signed certificates.
+
+This hybrid approach is needed because the Controller can be configured in two ways:
+
+* *Controller with a publicly valid certificate*
+    * can be configured manually by passing `--controller-cert` and `--controller-key` command-line arguments to `orchard controller run`
+* *Controller with a self-signed certificate*
+    * configured automatically on first Controller start-up when no `--controller-cert` and `--controller-key` command-line arguments are passed
+
+Below we'll explain how Orchard client and Worker secure the connection when accessing these two Controller types.
+
+### Client
+
+Client is associated with the Controller using a `orchard context create` command, which works as follows:
+
+* Client attempts to connect to the Controller and validate its certificate using host's root CA set (can be disabled with [`--no-pki`](#--no-pki-override))
+* if the Client encounters a  *Controller with a publicly valid certificate*, that would be the last step and the association would succeed
+* if the Client is dealing with *Controller with a self-signed certificate*, the Client will do another connection attempt to probe the Controller's certificate
+* the probed Controller's certificate fingerprint is then presented to the user, and if the user agrees to trust it, the Client then considers that certificate to be trusted for a given context
+* Client finally connects to the Controller again with a trusted CA set containing only that certificate, executes the final API sanity checks, and if everything is OK then the association succeeds
+
+Afterward, each interaction with the Controller  (e.g. `orchard create vm` command) will stick to the chosen verification method and will re-verify the presented Controller's certificate against:
+
+* *Controller with a self-signed certificate*: a trusted certificate stored in the Orchard's configuration file
+* *Controller with a publicly valid certificate*: host's root CA set
+
+### Worker
+
+To make the Worker connect to the Controller, a Bootstrap Token needs to be obtained using the `orchard get bootstrap-token` command.
+
+While this approach provides a less ad-hoc experience than that you'd have with `orchard context create`, it allows one to mass-deploy workers non-interactively, using tools such as Ansible.
+
+This resulting Bootstrap Token will either include the Controller's certificate (when the current context is with a *Controller with a self-signed certificate*) or omit it (when the current context is with a *Controller with a publicly valid certificate*).
+
+The way Worker connects to the Controller using the `orchard worker run` command is as follows:
+
+* when the Bootstrap Token contains the Controller's certificate:
+    * the Orchard Worker will try to connect to the Controller with a trusted CA set containing only that certificate
+* when the Bootstrap Token has no Controller's certificate:
+    * the Orchard Worker will try the PKI approach (can be disabled with [`--no-pki`](#--no-pki-override) to effectively prevent the Worker from connecting) and fail if certificate verification using PKI is not possible
+
+### `--no-pki` override
+
+If you only intend to access the *Controller with a self-signed certificate* and want to additionally guard yourself against [CA compromises](https://en.wikipedia.org/wiki/Certificate_authority#CA_compromise) and other PKI-specific attacks, pass a `--no-pki` command-line argument to the following commands:
+
+* `orchard context create --no-pki`
+    * this will prevent the Client from using PKI and will let you interactively verify the Controller's certificate fingerprint before connecting, thus creating a non-PKI association
+* `orchard worker run --no-pki`
+    * this will prevent the Worker from trying to use PKI when connecting to the Controller using a Bootstrap Token that has no certificate included in it, thus failing fast and letting you know that you need to create a proper Bootstrap Token
+
+We've deliberately chosen not to use environment variables (e.g. `ORCHARD_NO_PKI`) because they fail silently (e.g. due to a typo), compared to command-line arguments, which will result in an error that is much easier to detect.
@@ -0,0 +1,195 @@
+## Introduction
+
+Compared to Worker, which can only be deployed on a macOS machine, Controller can be also deployed on Linux.
+
+In fact, we've made a [container image](https://github.com/cirruslabs/orchard/pkgs/container/orchard) to ease deploying the Controller in container-native environments such as Kubernetes.
+
+Another thing to keep in mind that Orchard API is secured by default: all requests must be authenticated with the credentials of a service account. When you first run Orchard Controller, a `bootstrap-admin` service account will be created automatically and credentials will be printed to the standard output.
+
+If you already have a token in mind that you want to use for the `bootstrap-admin` service account, or you've got locked out and want this service account with a well-known password back, you can set the `ORCHARD_BOOTSTRAP_ADMIN_TOKEN` when running the controller.
+
+For example to use a secure, random value:
+
+```bash
+ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$(openssl rand -hex 32) orchard controller run
+```
+
+## Deployment Methods
+
+While you can always start `orchard controller run` manually with the required arguments, this method is not recommended due to lack of persistence.
+
+In the following sections you'll find several examples of how to run Orchard Controller in various environments in a more persistent way. Feel free to submit PRs with more examples.
+
+### Google Compute Engine
+
+An example below will deploy a single instance of Orchard Controller in Google Cloud Compute Engine in `us-central1` region.
+
+First, let's create a static IP address for our instance:
+
+```bash
+gcloud compute addresses create orchard-ip --region=us-central1
+export ORCHARD_IP=$(gcloud compute addresses describe orchard-ip --format='value(address)' --region=us-central1)
+```
+
+Once we have the IP address, we can create a new instance with Orchard Controller running inside a container:
+
+```bash
+gcloud compute instances create-with-container orchard-controller \
+  --machine-type=e2-micro \
+  --zone=us-central1-a \
+  --image-family cos-stable \
+  --image-project cos-cloud \
+  --tags=https-server \
+  --address=$ORCHARD_IP \
+  --container-image=ghcr.io/cirruslabs/orchard:latest \
+  --container-env=PORT=443 \
+  --container-env=ORCHARD_BOOTSTRAP_ADMIN_TOKEN=$ORCHARD_BOOTSTRAP_ADMIN_TOKEN \
+  --container-mount-host-path=host-path=/home/orchard-data,mode=rw,mount-path=/data
+```
+
+Now you can create a new context for your local client:
+
+```bash
+orchard context create --name production \
+  --service-account-name bootstrap-admin \
+  --service-account-token $ORCHARD_BOOTSTRAP_ADMIN_TOKEN \
+  https://$ORCHARD_IP:443
+```
+
+And select it as the default context:
+
+```bash
+orchard context default production
+```
+
+### Kubernetes (GKE, EKS, etc.)
+
+The easiest way to run Orchard Controller on Kubernetes is to expose it through the `LoadBalancer` service.
+
+This way no fiddling with the TLS certificates and HTTP proxying is needed, and most cloud providers will allocate a ready-to-use IP-address that can directly used in `orchard context create` and `orchard worker run` commands, or additionally assigned to a DNS domain name for a more memorable hostname.
+
+Do deploy on Kubernetes, only three resources are needed:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: orchard-controller
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+  # Uncomment this when deploying on Amazon's EKS and
+  # change to the desired storage class name if needed
+  # storageClassName: gp2
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: orchard-controller
+spec:
+  serviceName: orchard-controller
+  replicas: 1
+  selector:
+    matchLabels:
+      app: orchard-controller
+  template:
+    metadata:
+      labels:
+        app: orchard-controller
+    spec:
+      containers:
+        - name: orchard-controller
+          image: ghcr.io/cirruslabs/orchard:latest
+          volumeMounts:
+            - mountPath: /data
+              name: orchard-controller
+      volumes:
+        - name: orchard-controller
+          persistentVolumeClaim:
+            claimName: orchard-controller
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: orchard-controller
+spec:
+  selector:
+    app: orchard-controller
+  ports:
+    - protocol: TCP
+      port: 6120
+      targetPort: 6120
+  type: LoadBalancer
+```
+
+Once deployed, the bootstrap credentials will be printed to the standard output. You can inspect them by running `kubectl logs deployment/orchard-controller`.
+
+The resources above ensure that Controller's database is stored in a persistent storage and survives restats.
+
+You can further allocate a static IP address and use it by adding annotations to the `Service` resource. Here's how to do that:
+
+* on Google's GKE: <https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer-parameters#spd-static-ip>
+* on Amazon's EKS: <https://kubernetes.io/docs/reference/labels-annotations-taints/#service-beta-kubernetes-io-aws-load-balancer-eip-allocations>
+
+### systemd service on Debian-based distributions
+
+This should work for most Debian-based distributions like Debian, Ubuntu, etc.
+
+Firstly, make sure that the APT transport for downloading packages via HTTPS and common X.509 certificates are installed:
+
+```shell
+sudo apt-get update && sudo apt-get -y install apt-transport-https ca-certificates
+```
+
+Then, add the Cirrus Labs repository:
+
+```shell
+echo "deb [trusted=yes] https://apt.fury.io/cirruslabs/ /" | sudo tee /etc/apt/sources.list.d/cirruslabs.list
+```
+
+Update the package index files and install the Orchard Controller:
+
+```shell
+sudo apt-get update && sudo apt-get -y install orchard-controller
+```
+
+Finally, enable and start the Orchard Controller systemd service:
+
+```shell
+sudo systemctl enable orchard-controller
+sudo systemctl start orchard-controller
+```
+
+The bootstrap credentials will be printed to the standard output. You can inspect them by running `sudo systemctl status orhcard-controller` or `journalctl -u orchard-controller`.
+
+### systemd service on RPM-based distributions
+
+This should work for most RPM-based distributions like Fedora, CentOS, etc.
+
+First, create a `/etc/yum.repos.d/cirruslabs.repo` file with the following contents:
+
+```ini
+[cirruslabs]
+name=Cirrus Labs Repo
+baseurl=https://yum.fury.io/cirruslabs/
+enabled=1
+gpgcheck=0
+```
+
+Then, install the Orchard Controller:
+
+```shell
+sudo yum -y install orchard-controller
+```
+
+Finally, enable and start the Orchard Controller systemd service:
+
+```shell
+systemctl enable orchard-controller
+systemctl start orchard-controller
+```
+
+The bootstrap credentials will be printed to the standard output. You can inspect them by running `sudo systemctl status orhcard-controller` or `journalctl -u orchard-controller`.
@@ -0,0 +1,127 @@
+## Obtain a Boostrap Token
+
+First, create a service account with a minimal set of roles (`compute:read` and `compute:write`) required for proper Worker functioning:
+
+```bash
+orchard create service-account worker-pool-m1 --roles "compute:read" --roles "compute:write"
+```
+
+Then, generate a Bootstrap Token for this service account:
+
+```shell
+orchard get bootstrap-token worker-pool-m1
+```
+
+We will reference the value of the Bootstrap Token generated here as `${BOOTSTRAP_TOKEN}` below.
+
+Further, we assume that Orchard controller is available on `orchard.example.com`
+
+## Deployment Methods
+
+While you can always run `orchard worker run` manually with the required arguments, this method of deploying the Worker is not recommended.
+
+Instead, we've listed a more persistent methods of a Worker deployment below.
+
+### launchd
+
+[launchd](https://launchd.info/) is an init system for macOS that manages daemons, agents and other background processes.
+
+In this deployment method, we'll create a new job definition file for the launchd to manage on its behalf.
+
+To begin, first install Orchard:
+
+```shell
+brew install cirruslabs/cli/orchard
+```
+
+Ensure that the following command:
+
+```shell
+which orchard
+```
+
+...yields `/opt/homebrew/bin/orchard`. If not, you'll need to replace all of the occurences of `/opt/homebrew/bin/orchard` in the job definition below.
+
+Then, create a launchd job definition in `/Library/LaunchDaemons/org.cirruslabs.orchard.worker.plist` with the following contents:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+  <dict>
+    <key>Label</key>
+    <string>org.cirruslabs.orchard.worker</string>
+    <key>UserName</key>
+    <string>admin</string>
+    <key>Program</key>
+    <string>/opt/homebrew/bin/orchard</string>
+    <key>ProgramArguments</key>
+    <array>
+      <string>/opt/homebrew/bin/orchard</string>
+      <string>worker</string>
+      <string>run</string>
+      <string>--bootstrap-token</string>
+      <string>${BOOTSTRAP_TOKEN}</string>
+      <string>orchard.example.com</string>
+    </array>
+    <key>EnvironmentVariables</key>
+    <dict>
+      <key>PATH</key>
+      <string>/bin:/usr/bin:/usr/local/bin:/opt/homebrew/bin</string>
+    </dict>
+    <key>WorkingDirectory</key>
+    <string>/var/empty</string>
+    <key>RunAtLoad</key>
+    <true/>
+    <key>KeepAlive</key>
+    <true/>
+    <key>StandardOutPath</key>
+    <string>/Users/admin/orchard-launchd.log</string>
+    <key>StandardErrorPath</key>
+    <string>/Users/admin/orchard-launchd.log</string>
+  </dict>
+</plist>
+```
+
+This assumes that your macOS user on the host is named `admin`. If not, change all occurrences of `admin` in the job definition above to `$USER`.
+
+Finally, change the `orchard.example.com` to the FQDN or an IP-address of your Orchard Controller.
+
+Now, you can start the job:
+
+```shell
+launchctl load -w /Library/LaunchDaemons/org.cirruslabs.orchard.worker.plist
+```
+
+### Ansible
+
+If you have a set of machines that you want to use as Orchard Workers, you can use [Ansible](https://docs.ansible.com/) to configure them.
+
+We've created the [cirruslabs/ansible-orchard](https://github.com/cirruslabs/ansible-orchard) repository with a basic Ansible playbook for convenient setup.
+
+To use it, clone it locally:
+
+```shell
+git clone https://github.com/cirruslabs/ansible-orchard.git
+cd ansible-orchard/
+```
+
+Make sure that the Ansible Galaxy dependencies are installed:
+
+```shell
+ansible-galaxy install -r requirements.yml
+```
+
+Then, edit the `production-pool` file and populate the following fields:
+
+* `hosts` — replace `worker-1.hosts.internal` with your worker FQDN or IP-address and add more hosts if needed
+* `ansible_user` — set it macOS user on the host for the SSH to work
+* `orchard_worker_user` — set it macOS user on the host under which the Worker will run, e.g. `admin`
+* `orchard_worker_controller_url` — set it to FQDN or an IP-address of your Orchard Controller, for example, `orchard.example.com`
+* `orchard_worker_bootstrap_token` — set it to `${BOOTSTRAP_TOKEN}` we've generated above
+
+Deploy the playbook:
+
+```shell
+ansible-playbook --inventory-file production-pool --ask-pass playbook-workers.yml
+```
@@ -0,0 +1,187 @@
+Orchard has a REST API that follows [OpenAPI specification](https://swagger.io/specification/) and is described in [`api/openapi.yaml`](https://github.com/cirruslabs/orchard/blob/main/api/openapi.yaml).
+
+You can run `orchard dev` locally and navigate to `http://127.0.0.1:6120/v1/` for interactive documentation.
+
+![](/assets/images/orchard/orchard-api-documentation-browser.png)
+
+## Using the API
+
+Below you'll find examples of using Orchard API via vanilla Python's request library and Golang package that Orchard CLI build on top of.
+
+### Authentication
+
+When running in non-development mode, Orchard API expects a [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) to be provided for each API call.
+
+Below you'll find two snippets that retrieve controller's information and output its version:
+
+#### Authentication in Python
+
+```python
+import requests
+from requests.auth import HTTPBasicAuth
+
+
+def main():
+  # Authentication
+  basic_auth = HTTPBasicAuth("service account name", "service account token")
+
+  response = requests.get("http://127.0.0.1:6120/v1/info", auth=basic_auth)
+
+  print(response.json()["version"])
+
+
+if __name__ == '__main__':
+  main()
+```
+
+#### Authentication in Golang
+
+```go
+package main
+
+import (
+  "context"
+  "fmt"
+  "github.com/cirruslabs/orchard/pkg/client"
+  "log"
+)
+
+func main() {
+  client, err := client.New()
+  if err != nil {
+    log.Fatalf("failed to initialize Orchard API client: %v", err)
+  }
+
+  controllerInfo, err := client.Controller().Info(context.Background())
+  if err != nil {
+    log.Fatalf("failed to retrieve controller's information: %v", err)
+  }
+
+  fmt.Println(controllerInfo.Version)
+}
+```
+
+Note that we don't provide any credentials for Golang's version of the snippet: this is because Orchard's Golang API client (`github.com/cirruslabs/orchard/pkg/client`) has the ability to read the current's user Orchard context automatically.
+
+### Creating a VM
+
+A more intricate example would be spinning off a VM with a startup script that outputs date, reading its logs and removing it from the controller:
+
+#### Creating a VM in Python
+
+```python
+import time
+import uuid
+
+import requests
+from requests.auth import HTTPBasicAuth
+
+
+def main():
+  vm_name = str(uuid.uuid4())
+
+  basic_auth = HTTPBasicAuth("service account name", "service account token")
+
+  # Create VM
+  response = requests.post("http://127.0.0.1:6120/v1/vms", auth=basic_auth, json={
+    "name": vm_name,
+    "image": "ghcr.io/cirruslabs/macos-sonoma-base:latest",
+    "cpu": 4,
+    "memory": 4096,
+    "startup_script": {
+      "script_content": "date",
+    }
+  })
+  response.raise_for_status()
+
+  # Retrieve VM's logs
+  while True:
+    response = requests.get(f"http://127.0.0.1:6120/v1/vms/{vm_name}/events", auth=basic_auth)
+    response.raise_for_status()
+
+    result = response.json()
+
+    if isinstance(result, list) and len(result) != 0:
+      print(result[0]["payload"])
+      break
+
+    time.sleep(1)
+
+  # Delete VM
+  response = requests.delete(f"http://127.0.0.1:6120/v1/vms/{vm_name}", auth=basic_auth)
+  response.raise_for_status()
+
+
+if __name__ == '__main__':
+  main()
+```
+
+#### Creating a VM in Golang
+
+```go
+package main
+
+import (
+  "context"
+  "fmt"
+  "github.com/cirruslabs/orchard/pkg/client"
+  v1 "github.com/cirruslabs/orchard/pkg/resource/v1"
+  "github.com/google/uuid"
+  "log"
+  "time"
+)
+
+func main() {
+  vmName := uuid.New().String()
+
+  client, err := client.New()
+  if err != nil {
+    log.Fatalf("failed to initialize Orchard API client: %v", err)
+  }
+
+  // Create VM
+  err = client.VMs().Create(context.Background(), &v1.VM{
+    Meta: v1.Meta{
+      Name: vmName,
+    },
+    Image:  "ghcr.io/cirruslabs/macos-sonoma-base:latest",
+    CPU:    4,
+    Memory: 4096,
+    StartupScript: &v1.VMScript{
+      ScriptContent: "date",
+    },
+  })
+  if err != nil {
+    log.Fatalf("failed to create VM: %v")
+  }
+
+  // Retrieve VM's logs
+  for {
+    vmLogs, err := client.VMs().Logs(context.Background(), vmName)
+    if err != nil {
+      log.Fatalf("failed to retrieve VM logs")
+    }
+
+    if len(vmLogs) != 0 {
+      fmt.Println(vmLogs[0])
+      break
+    }
+
+    time.Sleep(time.Second)
+  }
+
+  // Delete VM
+  if err := client.VMs().Delete(context.Background(), vmName); err != nil {
+    log.Fatalf("failed to delete VM: %v", err)
+  }
+}
+```
+
+## Resource management
+
+Some resources, such as `Worker` and `VM`, have a `resource` field which is a dictionary that maps between resource names and their amounts (amount requested or amount provided, depending on the resource) and is useful for scheduling.
+
+Well-known resources:
+
+* `org.cirruslabs.tart-vms` — number of Tart VM slots available on the machine or requested by the VM
+    * this number is `2` for workers and `1` for VMs by default
@@ -0,0 +1,30 @@
+## Backups
+
+In order to backup the Orchard Controller, simply copy its `ORCHARD_HOME` (which defaults to `~/.orchard/`) directory somewhere safe and restore it when needed.
+
+This directory contains a BadgerDB database that Controller uses to store state and an X.509 certificate with key.
+
+## Upgrades
+
+Since the Orchard's initial release, we've managed to maintain the backwards compatibility between versions up to this day, so generally, it doesn't matter whether you upgrade the Controller or Worker(s) first.
+
+In case a new functionality is introduced, you might be required to finish the upgrade of both the Controller and the Worker(s) to be able to use it fully.
+
+In case there will be backwards-incompatible changes introduced in the future, we will try to do our best and highlight this in the [release notes](https://github.com/cirruslabs/orchard/releases) accordingly.
+
+## Observability
+
+Both the Controller and Worker produce some useful OpenTelemetry metrics. Metrics are scoped with `org.cirruslabs.orchard` prefix and include information about resource utilization, statuses or Workers, scheduling/pull time and many more.
+
+By default, the telemetry is sent to `https://localhost:4317` using the gRPC protocol and to `http://localhost:4318` using the HTTP protocol.
+
+You can override this by setting the [standard OpenTelemetry environment variable](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/) `OTEL_EXPORTER_OTLP_ENDPOINT`.
+
+Please refer to [OTEL Collector documentation](https://opentelemetry.io/docs/collector/) for instruction on how to setup a sidecar for the metrics collections or find out if your SaaS monitoring has an available OTEL endpoint (see [Honeycomb](https://docs.honeycomb.io/send-data/opentelemetry/) as an example).
+
+### Sending metrics to Google Cloud Platform
+
+There are two standard options of ingesting metrics procuded by Orchard Controller and Workers into the GCP:
+
+* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) + [Google Cloud Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/googlecloudexporter/README.md) — open-source solution that can be later re-purposed to send metrics to any OTLP-compatible endpoint by swapping a single [exporter](https://opentelemetry.io/docs/collector/configuration/#exporters)
+* [Ops Agent](https://cloud.google.com/monitoring/agent/ops-agent/otlp) — Google-backed solution with a syntax similar to OpenTelemetry Collector, but tied to GCP-only
@@ -0,0 +1,101 @@
+Tart is great for running workloads on a single machine, but what if you have more than one computer at your disposal
+and
+a couple of VMs is not enough anymore for your needs? This is where [Orchard](https://github.com/cirruslabs/orchard)
+comes in to play!
+
+It allows you to orchestrate multiple Tart-capable hosts from either an Orchard CLI (which we demonstrate below)
+or [through the API](/orchard/integration-guide).
+
+The easiest way to start is to run Orchard in local development mode:
+
+```shell
+brew install cirruslabs/cli/orchard
+orchard dev
+```
+
+This will run an Orchard Controller and an Orchard Worker in a single process on your local machine, allowing you to
+test both the CLI functionality and the API from a tool like cURL or programming language of choice, without the need to
+authenticate requests.
+
+Note that in production deployments, these two components are started separately and enable security by default. Please
+refer to [Deploying Controller](/orchard/deploying-controller) and [Deploying Workers](/orchard/deploying-workers) for
+more information.
+
+## Creating Virtual Machines
+
+Now, let's create a Virtual Machine:
+
+```shell
+orchard create vm --image ghcr.io/cirruslabs/macos-sonoma-base:latest sonoma-base
+```
+
+You can check a list of VM resources to see if the Virtual Machine we've created above is already running:
+
+```shell
+orchard list vms
+```
+
+## Accessing Virtual Machines
+
+Orchard has an ability to do port forwarding that `ssh` and `vnc` commands are built on top of. All port forwarding
+connections are done via the Orchard Controller instance which "proxies" a secure connection to the Orchard Workers.
+
+Therefore, your workers can be located under a stricter firewall that only allows connections to the Orchard Controller
+instance. Orchard Controller instance is secured by default and all API calls are authenticated and authorized.
+
+### SSH
+
+To SSH into a VM, use the `orchard ssh` command:
+
+```shell
+orchard ssh vm sonoma-base
+```
+
+You can specify the `--username` and `--password` flags to specify the username/password pair to use for the SSH
+protocol. By default, `admin`/`admin` is used.
+
+You can also execute remote commands instead of spawning a login shell, similarly to how OpenSSH's `ssh` command accepts
+a command argument:
+
+```shell
+orchard ssh vm sonoma-base "uname -a"
+```
+
+You can execute scripts remotely this way, by telling the remote command-line interpreter to read from the standard
+input and using the redirection operator as follows:
+
+```shell
+orchard ssh vm sonoma-base "bash -s" < script.sh
+```
+
+### VNC
+
+Similarly to `ssh` command, you can use `vnc` command to open Screen Sharing into a remote VM:
+
+```shell
+orchard vnc vm sonoma-base
+```
+
+You can specify the `--username` and `--password` flags to specify the username/password pair to use for the VNC
+protocol. By default, `admin`/`admin` is used.
+
+## Deleting Virtual Machines
+
+The following command will delete the VM we've created above and clean-up the resources associated with it:
+
+```shell
+orchard delete vm sonoma-base
+```
+
+## Environment variables
+
+In addition to controlling the Orchard via the CLI arguments, there are environment variables that may be beneficial
+both when automating Orchard and in daily use:
+
+| Variable name                   | Description                                                                                                                                                                                                                                                                                                                                                                  |
+|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `ORCHARD_HOME`                  | Override Orchard's home directory. Useful when running multiple Orchard instances on the same host and when testing.                                                                                                                                                                                                                                                         |
+| `ORCHARD_LICENSE_TIER`          | The default license limit only allows connecting 4 Orchard Workers to the Orchard Controller. If you've purchased a [Gold Tier License](/licensing/), set this variable to `gold` to increase the limit to 20 Orchard Workers. And if you've purchased a [Platinum Tier License](/licensing/), set this variable to `platinum` to increase the limit to 200 Orchard Workers. |
+| `ORCHARD_URL`                   | Override controller URL on per-command basis.                                                                                                                                                                                                                                                                                                                                |
+| `ORCHARD_SERVICE_ACCOUNT_NAME`  | Override service account name (used for controller API auth) on per-command basis.                                                                                                                                                                                                                                                                                           |
+| `ORCHARD_SERVICE_ACCOUNT_TOKEN` | Override service account token (used for controller API auth) on per-command basis.                                                                                                                                                                                                                                                                                          |
@@ -85,10 +85,10 @@
      <!-- landing image -->
      <div class="tx-landing__image">
        <dotlottie-player
-          src="/assets/animations/TartLogo.lottie"
-          mode="normal"
-          style="width: 75%; margin: auto"
-          autoplay
+            src="/assets/animations/TartLogo.lottie"
+            mode="normal"
+            style="width: 75%; margin: auto"
+            autoplay
        />
      </div>

@@ -114,10 +114,9 @@
      <header class="md-typeset">
        <h1 id="virtualization-and-beyond">
          Virtualization and beyond
-          <a
-            href="#virtualization-and-beyond"
-            class="headerlink"
-            title="Permanent link"
+          <a href="#virtualization-and-beyond"
+             class="headerlink"
+             title="Permanent link"
          >
            ¶
          </a>
@@ -125,12 +124,11 @@
      </header>
      <div class="mdx-spotlight">
        <figure class="mdx-spotlight__feature">
-          <img
-            src="assets/images/spotlight/virtualization-framework.webp"
-            alt="Apple’s native Virtualization.Framework"
-            loading="lazy"
-            width="500"
-            height="212"
+          <img src="assets/images/spotlight/virtualization-framework.webp"
+               alt="Apple’s native Virtualization.Framework"
+               loading="lazy"
+               width="500"
+               height="212"
          />
          <figcaption class="md-typeset">
            <h2>Native performance</h2>
@@ -144,12 +142,11 @@
          </figcaption>
        </figure>
        <figure class="mdx-spotlight__feature">
-          <img
-            src="assets/images/spotlight/supported-registries.webp"
-            alt="OCI-compatible container registries"
-            loading="lazy"
-            width="500"
-            height="160"
+          <img src="assets/images/spotlight/supported-registries.webp"
+               alt="OCI-compatible container registries"
+               loading="lazy"
+               width="500"
+               height="160"
          />
          <figcaption class="md-typeset">
            <h2>Remote storage for Virtual Machines</h2>
@@ -161,39 +158,39 @@
          </figcaption>
        </figure>
        <figure class="mdx-spotlight__feature">
-          <img
-            src="assets/images/spotlight/github-actions-runners.webp"
-            alt="GitHub Actions Runners"
-            loading="lazy"
-            width="500"
-            height="280"
+          <img src="assets/images/spotlight/github-actions-runners.webp"
+               alt="GitHub Actions Runners"
+               loading="lazy"
+               width="500"
+               height="280"
          />
          <figcaption class="md-typeset">
            <h2>Seamless integration with your existing automations</h2>
            <p>
              Tart integrates with many continuous integration systems, including a dedicated
              service of on-demand GitHub Actions Runners. With a single line change, you can cut your
-              CI/CD costs by up to <b>30 times</b> by using <a href="https://cirrus-runners.app/">Cirrus Runners</a>
+              CI/CD costs by up to <b>30 times</b> by using <a href="https://cirrus-runners.app/">Cirrus
+              Runners</a>
              to run your workflows.
            </p>
          </figcaption>
        </figure>
        <figure class="mdx-spotlight__feature">
          <div id="lottie-player">
-            <dotlottie-player
-              src="/assets/animations/Orchard.lottie"
-              mode="normal"
-              style="height: 280px; margin: auto"
-              autoplay
-              loop
+            <dotlottie-player src="/assets/animations/Orchard.lottie"
+                              mode="normal"
+                              style="height: 280px; margin: auto"
+                              autoplay
+                              loop
            />
          </div>
          <figcaption class="md-typeset">
            <h2>Run at scale with <a href="https://github.com/cirruslabs/orchard">Orchard</a></h2>
            <p>
-              Tart toolset includes Orchard Orchestration &mdash; tool to run and manage Tart virtual machines
-              at scale on a cluster of Apple Silicon hosts. An Orchard Cluster exposes a simple REST API to manage
-              thousands virtual machines. Orchard CLI allows accessing remote virtual machines like they run locally.
+              Tart toolset includes Orchard Orchestration &mdash; tool to run and manage Tart virtual
+              machines at scale on a cluster of Apple Silicon hosts. An Orchard Cluster exposes a simple REST API to
+              manage thousands virtual machines. Orchard CLI allows accessing remote virtual machines like they run
+              locally.
            </p>
          </figcaption>
        </figure>
@@ -208,38 +205,38 @@
      <header class="md-typeset">
        <h1 id="powerhouse">
          Automation Powerhouse
-          <a
-            href="#powerhouse"
-            class="headerlink"
-            title="Permanent link"
+          <a href="#powerhouse"
+             class="headerlink"
+             title="Permanent link"
          >
            ¶
          </a>
        </h1>
      </header>
      <script>
-        fetch("https://api.github.com/repos/cirruslabs/tart/releases?per_page=100")
-          .then((response) => response.json())
-          .then((releases) => {
-            let allDownloads = 0;
-            for (let release of releases) {
-              for (let asset of release.assets) {
-                if (asset && asset.content_type === "application/octet-stream") {
-                  allDownloads += asset.download_count || 0
-                }
-              }
-            }
-            let counterElement = document.getElementById('installation-counter');
-            if (counterElement) {
-              // Live installation count is available starting version 1.0.0
-              // Prior Tart was installed a little over 14,000 times, let's count them too
-              let installationPriorV1 = 14
-              counterElement.textContent = (installationPriorV1 + Math.round(allDownloads / 1000)) + ",000"
-            }
-          })
+          fetch("https://api.github.com/repos/cirruslabs/tart/releases?per_page=100")
+              .then((response) => response.json())
+              .then((releases) => {
+                  let allDownloads = 0;
+                  for (let release of releases) {
+                      for (let asset of release.assets) {
+                          if (asset && asset.content_type === "application/octet-stream") {
+                              allDownloads += asset.download_count || 0
+                          }
+                      }
+                  }
+                  let counterElement = document.getElementById('installation-counter');
+                  if (counterElement) {
+                      // Live installation count is available starting version 1.0.0
+                      // Prior Tart was installed a little over 14,000 times, let's count them too
+                      let installationPriorV1 = 14
+                      counterElement.textContent = (installationPriorV1 + Math.round(allDownloads / 1000)) + ",000"
+                  }
+              })
      </script>
      <h2>
-        With more than <strong id="installation-counter">25,000</strong> installations to date, Tart has been adopted for various scenarios.
+        With more than <strong id="installation-counter">25,000</strong> installations to date, Tart has been
+        adopted for various scenarios.
        Its applications range from powering CI/CD pipelines and reproducible local development environments,
        to helping in the testing of device management systems without actual physical devices.
      </h2>
@@ -254,10 +251,9 @@
      <header class="md-typeset">
        <h1 id="what-our-users-say">
          What our users say
-          <a
-            href="#what-our-users-say"
-            class="headerlink"
-            title="Permanent link"
+          <a href="#what-our-users-say"
+             class="headerlink"
+             title="Permanent link"
          >
            ¶
          </a>
@@ -265,70 +261,60 @@
      </header>
      <div class="mdx-users">
        <figure class="mdx-users__testimonial">
-          <img
-            src="assets/images/users/mitchell-hashimoto.webp"
-            alt="Mitchell Hashimoto"
-            loading="lazy"
-            width="200"
-            height="200"
+          <img src="assets/images/users/mikhail-tokarev.webp"
+               alt="Mikhail Tokarev"
+               loading="lazy"
+               width="200"
+               height="200"
          />
          <figcaption class="md-typeset">
-            <h2>Mitchell Hashimoto</h2>
            <h3>
-              <a href="https://www.hashicorp.com/" target="_blank">HashiCorp</a> co-founder
+              Mikhail Tokarev, CTO at <a href="https://codemagic.io/start/" target="_blank">Codemagic</a>
            </h3>
            <hr/>
            <cite>
-              I've been using "Cirrus Runners" since <a href="https://x.com/mitchellh/status/1731071326201561194" target="_blank">that tweet</a> and
-              it has been fantastic. Huge speed increase, huge cost decrease, zero maintenance, exactly what I wanted.
+              Thanks to the minimal overhead of using the Apple Virtualization
+              API, we’ve seen some performance improvements in booting new
+              virtual machines compared with Anka.
            </cite>
          </figcaption>
        </figure>
        <figure class="mdx-users__testimonial">
-          <img
-            src="assets/images/users/seb-jachec.webp"
-            alt="Sebastian Jachec"
-            loading="lazy"
-            width="200"
-            height="200"
+          <img src="assets/images/users/expo.webp"
+               alt="Expo"
+               loading="lazy"
+               width="200"
+               height="200"
          />
          <figcaption class="md-typeset">
-            <h2>Sebastian Jachec</h2>
            <h3>
-              Mobile Engineer at
-              <a href="https://daybridge.com/" target="_blank">Daybridge</a>
+              Infrastructure Team at <a href="https://expo.dev/" target="_blank">Expo</a>
            </h3>
            <hr/>
            <cite>
-              It&rsquo;s been plain-sailing with the
-              <a href="/integrations/github-actions">Cirrus Runners</a>&nbsp;&mdash;
-              they&rsquo;ve been great! They&rsquo;re consistently&nbsp;60+%
-              faster on&nbsp;workflows that we&nbsp;previously used Github
-              Actions&rsquo; macOS runners for.
+              Tart was the practical way for us to use the Virtualization framework. Cirrus Labs’
+              continued maintenance and support gives us confidence, and it is also important for us
+              to be able to read the source code when we need to understand an abstraction layer below.
            </cite>
          </figcaption>
        </figure>
        <figure class="mdx-users__testimonial">
-          <img
-            src="assets/images/users/max-lapides.webp"
-            alt="Max Lapides"
-            loading="lazy"
-            width="200"
-            height="200"
+          <img src="assets/images/users/snowflake.webp"
+               alt="Snowflake"
+               loading="lazy"
+               width="200"
+               height="200"
          />
          <figcaption class="md-typeset">
-            <h2>Max Lapides</h2>
            <h3>
-              Senior Mobile Engineer at
-              <a href="https://www.tonal.com/" target="_blank">Tonal</a>
+              Red Team at <a href="https://www.snowflake.com/" target="_blank">Snowflake</a>
            </h3>
            <hr/>
            <cite>
-              Previously, we were using the GitHub&#8209;hosted macOS runners
-              and our iOS build took ~30&nbsp;minutes. Now with
-              <a href="/integrations/github-actions">Cirrus Runners</a>, the iOS build only
-              takes ~12&nbsp;minutes. That’s a huge boost to our productivity,
-              and for only $150/month per runner it is much less expensive too.
+              The Snowflake Red Team had a need for macOS CI/CD and a segmented macOS development
+              environment. We solved this problem and shared our implementation with macOS EC2 and Tart.
+              We also automated this process with Terraform/Packer to simplify the deployment of our
+              infrastructure and machine images.
            </cite>
          </figcaption>
        </figure>
@@ -91,13 +91,19 @@ nav:
  - "Home": index.md
  - "Quick Start": quick-start.md
  - "Integrations":
-    - "Self-hosted CI": integrations/cirrus-cli.md
-    - "GitHub Actions": https://cirrus-runners.app/
-    - "GitLab Runner": integrations/gitlab-runner.md
-    - "Buildkite": integrations/buildkite.md
-    - "Managing VMs": integrations/vm-management.md
+      - "Self-hosted CI": integrations/cirrus-cli.md
+      - "GitHub Actions": https://cirrus-runners.app/
+      - "GitLab Runner": integrations/gitlab-runner.md
+      - "Buildkite": integrations/buildkite.md
+      - "Managing VMs": integrations/vm-management.md
  - "Support & Licensing": licensing.md
-  - "Orchestration": https://github.com/cirruslabs/orchard
+  - "Orchestration":
+      - "Quick Start": orchard/quick-start.md
+      - "Architecture and Security": orchard/architecture-and-security.md
+      - "Deploying Controller": orchard/deploying-controller.md
+      - "Deploying Workers": orchard/deploying-workers.md
+      - "Managing the Cluster": orchard/managing-cluster.md
+      - "Integrating with the API": orchard/integration-guide.md
  - "FAQ": faq.md
  - "Legal":
      - 'Terms of Service': legal/terms.md
Author	SHA1	Message	Date
Nikolay Edigaryev	b1e88e1e51	tart run: do not remove "Edit" menu as its not present anymore (#946 )	2024-11-18 09:55:55 +01:00
Nikolay Edigaryev	b4de3bee83	tart pull: retry if we get URLError (#947 )	2024-11-15 23:14:47 +01:00
Fedor Korotkov	cd0f238a67	Allow to specify custom image in benchmarks (#941 )	2024-11-08 16:41:05 +00:00
Fedor Korotkov	02f94720c5	Set application category (#940 ) Was looking into performance and was wondering about Game Mode on Sonoma. This change is unrelated. Just found they have a category for tools like Tart.	2024-11-07 21:12:30 +00:00
Nikolay Edigaryev	c0443060cf	tart run: set "prohibited" activation policy when --no-graphics is set (#939 )	2024-11-07 15:20:09 -05:00
Fedor Korotkov	9c879b3f55	`tart run --nested` to enable nested virtualization when available (#938 ) Only works for Linux VMs under Sequoia hosts. Fixes #933 Fixes #701	2024-11-06 21:27:29 +04:00
Nikolay Edigaryev	f7b38769a9	tart pull: open the VM directory after pulling under a lock (#936 )	2024-11-05 00:01:14 +01:00
Nikolay Edigaryev	7c1ed4640f	Info.plist: do not use LSBackgroundOnly (#935 )	2024-11-04 19:08:37 +00:00
Nikolay Edigaryev	3fb8069edd	Linux VMs: do not use NVMe storage device (#932 )	2024-10-31 16:35:12 -04:00
Nikolay Edigaryev	c78c89e274	utimes(2): use errno to explain the error (#931 )	2024-10-31 16:33:03 -04:00
Fedor Korotkov	770220f905	Fixed plist file in version update (#927 )	2024-10-29 13:01:47 +04:00
Nikolay Edigaryev	768d1f9bad	PROFILING.md: document how to profile Tart using time(1) and xctrace(1) (#926 )	2024-10-28 18:49:10 +04:00
Fedor Korotkov	d49ed46439	Update access time on `pull` (#925 ) To make sure we won't prune then immediately after. Useful for when scenarios similar to Cirrus CLI when we make sure that several images are up-to-date before every request for task execution.	2024-10-25 23:33:30 +04:00
Nikolay Edigaryev	b52a857698	tart {clone,pull}: make deduplication opt-in (#924 )	2024-10-25 17:56:38 +04:00
Nikolay Edigaryev	3bf0bb22f3	CI: populate CFBundleShortVersionString in Info.plist (#923 )	2024-10-24 18:47:50 +00:00
Nikolay Edigaryev	accbd0cb33	Registry: prevent double authorization when getting a new token (#922 )	2024-10-23 23:51:55 +04:00
Nikolay Edigaryev	c0b20932c7	Prevent pipe deadlock when spawning a Process() (#916 )	2024-10-02 09:48:23 -04:00
Nikolay Edigaryev	3694af946c	Document automatic pruning in FAQ (#913 ) * Document automatic pruning in FAQ * tart {pull,clone}: consistent automatic pruning documentation in --help	2024-09-30 21:56:55 +04:00
Nikolay Edigaryev	dbf711a6c9	tart delete: return human-friendly error when local VM doesn't exist (#910 )	2024-09-26 14:45:14 +04:00
Nikolay Edigaryev	b9f24a40c1	Info.plist: set CFBundleName and CFBundleDisplayName to Tart (#909 )	2024-09-24 17:24:50 +04:00
Nikolay Edigaryev	10c6ace671	Re-generate ANTLR files using ANTLR 4.13.2 (#907 ) * Re-generate ANTLR files using ANTLR 4.13.2 * Package.swift: require exactly ANTLR of version 4.13.2	2024-09-20 17:50:14 +04:00
Nikolay Edigaryev	b98e23956b	Package.swift: bump Sentry SDK to 8.36.0 + upgrade other packages (#905 ) * Package.swift: bump Sentry SDK to 8.36.0 * $ swift package update	2024-09-19 19:06:37 +00:00
Fedor Korotkov	ce23f9c2a7	Completely disable audio devices in case of `--no-audio` (#904 ) This way VM won't have empty audio device at all. This should fix with an issue like that https://github.com/actions/runner-images/issues/9330	2024-09-17 09:40:37 +00:00
Nikolay Edigaryev	3da91e6518	tart run: provide a hint with names of other running VMs (#900 ) When VM limit gets exceeded.	2024-09-09 20:45:59 +04:00
Nikolay Edigaryev	7046886713	docs(orchard): document Kubernetes and systemd service deployment (#899 )	2024-09-09 16:40:17 +04:00
Nikolay EdigaryevandFedor Korotkov	3fde7d08dd	Orchard documentation (#897 ) * Orchard documentation * Fix typo Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com> * architecture-and-security.md: change list order --------- Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>	2024-08-28 00:09:57 +04:00
Fedor Korotkov	227301436c	Revert "Drop Monterey Support (#843 )" (#893 ) This reverts commit `017592075f`.	2024-08-14 14:57:55 -04:00
Nikolay Edigaryev	106eb5a2c8	tart push: re-try when encountering errors when pushing disk layers (#888 ) * tart push: re-try when encountering errors when pushing disk layers * Only re-try on URLError	2024-08-10 13:06:39 -04:00
Nikolay Edigaryev	10bf706653	tart push: avoid uploading blobs if they are already present (#887 ) By issuing HEAD requests to the registry before doing the actual upload.	2024-08-09 17:26:20 +04:00
Fedor Korotkov	ff928ad77d	Optimize DiskV2 Deduplication (#878 ) * Revert "Lowercase `tart.app` (#751)" This reverts commit `a9e2a19015`. * Optimize DiskV2 deduplication logic In case we cloned `disk.img` from a local image, check if data at offset has the expected contents already. * Hole punch only if needed * Calculate hash only if needed * subdataChunks optimization * Reapply "Lowercase `tart.app` (#751)" This reverts commit `e74e9c845a`. * format * Save at least 1GB on deduplication logic * Build separately * Revert "subdataChunks optimization" This reverts commit `e59382aeba`. * Another optimization * Removed debug log * reformat * Revert "Hole punch only if needed" This reverts commit `8c569fc5`	2024-08-05 12:24:31 -04:00
Nikolay Edigaryev	33b5cfe2ed	tart run: delay tilde (~) expansion until we're dealing with local path (#880 )	2024-08-05 15:42:10 +04:00
Nikolay Edigaryev	3892cdb00d	tart run: replace --sync with --root-disk-opts (#879 ) * VZDiskImageSynchronizationMode's "description" field is a dead code * Re-use the VZDiskImageSynchronizationMode extension * tart run: replace --sync with --root-disk-opts * VM: support root disk synchronization mode on macOS	2024-08-05 15:17:58 +04:00
Nicholas FitzRoy-Dale	5f2199ef3e	Support setting root disk synchronization mode (#875 ) * Support setting root disk synchronization mode Adds a new VMConfig parameter (tart get / tart set) called 'sync' which can be set to 'full' (default), 'fsync', or 'none', corresponding with the values of VZDiskImageSynchronizationMode and allowing a tradeoff between data integrity and speed. * Remove unused import * Fix formatting * Make root disk sync behaviour a commandline option	2024-08-05 13:12:43 +04:00
Fedor Korotkov	3f26baa341	Update testimonials to focus on Tart (#876 ) Cirrus Runners have their own testimonials and a website now. No need to mix things together.	2024-08-01 17:09:08 +04:00
Nikolay Edigaryev	06cae1296e	tart run: support disabling disk synchronization for --disk (#872 )	2024-07-25 20:15:07 +04:00
Nikolay EdigaryevandFedor Korotkov	1b81b12760	tart pull: try to re-use APFS blocks by cloning the base image (#864 ) * tart pull: try to re-use APFS blocks by cloning the base image * Punch a hole when a zero chunk is detected * Properly retrieve errno when hole punching operation fails * tart pull: do not retry on RuntimeError * Ensure that the holes we're about to punch are FS block size-aligned * VMDirectory: remove unused static variables * tart pull: log if we've found an image to deduplicate against * Do not prematurely read contents from disk * Only consider candidates with deduplicatedBytes more than 0 * APFS reuse UX/DX improvements (#870) * Show how much deduplication happening Improvement to the APFS deduplication logic which checks whether a disk image file `mayShareFileContent` with some other file, and then we put a custom attribute to track the deduplication since there is no way to get this information from APFS itself. It's not 100% accurate but given that OCI cache is immutable the actual disk usage can only be lover than that. * Use string attribute * Update Sources/tart/URL+Prunable.swift Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com> * Added SizeOnDisk colume --------- Co-authored-by: Nikolay Edigaryev <edigaryev@gmail.com> --------- Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>	2024-07-25 15:33:15 +00:00
Nikolay Edigaryev	4ed73bc775	--no-audio: only disable the source and sink (#869 ) To prevent crashes in the guest when playing or recording audio.	2024-07-18 15:10:44 +00:00
Nikolay Edigaryev	2dc25ce478	tart push: support --concurrency command-line argument (#868 ) * tart push: support --concurrency command-line argument * LayerizerTests: specify "concurrency" argument	2024-07-18 17:52:55 +04:00
Nikolay Edigaryev	1e74e268a5	DiskV2: change layer size to 512 * 1024 * 1024 bytes (#866 ) Needed to test https://github.com/cirruslabs/tart/pull/864.	2024-07-17 16:37:45 +00:00
Nikolay Edigaryev	bff344fb7f	tart login: better error when an improperly formatted host is provided (#863 ) * tart login: better error when an improperly formatted host is provided * Revert old behavior w.r.t. URLComponents()	2024-07-15 18:36:09 +04:00
Nikolay Edigaryev	ababe8cefc	tart pull: choose across multiple VM images to deduplicate against (#862 ) This is accomplished by saving the OCI VM image manifests on "tart pull" in "manifest.json" file and then using them on successive "tart pull"'s to find the best candidate that results in the most de-duplication, measured in bytes.	2024-07-15 18:36:01 +04:00
Fedor Korotkov	ea5313698e	Do not prune running VMs (#861 ) Also prevent pushing of a running VM Fixes #860	2024-07-15 07:10:51 -04:00
Fedor Korotkov	679289d7ab	Added Figma as a user (#858 ) See https://www.figma.com/open-source/ Plus reworked users section since now we can curate the best representative.	2024-07-10 15:06:01 +00:00