Rosetta support (#324)

Fixes #208

.cirrus.yml

@@ -1,13 +1,30 @@
 task:
   name: Test on Ventura
+  alias: test
   persistent_worker:
     labels:
-      name: Mac-Mini-M1
-  build_script: swift test
-  test_script: swift test
+      name: scaleway-m1
+  test_script:
+    - swift test
+  integration_test_script:
+    # Build Tart
+    - swift build
+    - codesign --sign - --entitlements Resources/tart.entitlements --force .build/debug/tart
+    - export PATH=$(pwd)/.build/arm64-apple-macosx/debug:$PATH
+    # Run integration tests
+    - cd integration-tests
+    - HOMEBREW_NO_AUTO_UPDATE=1 brew install virtualenv
+    - virtualenv venv
+    - source venv/bin/activate
+    - pip install -r requirements.txt
+    - pytest --verbose --junit-xml=pytest-junit.xml
+  pytest_junit_result_artifacts:
+    path: "integration-tests/pytest-junit.xml"
+    format: junit
 
 task:
   name: Build
+  alias: build
   only_if: $CIRRUS_TAG == ''
   macos_instance:
     image: ghcr.io/cirruslabs/macos-ventura-xcode:latest
@@ -19,6 +36,9 @@ task:
 task:
   name: Release
   only_if: $CIRRUS_TAG != ''
+  depends_on:
+    - test
+    - build
   macos_instance:
     image: ghcr.io/cirruslabs/macos-ventura-xcode:latest
   env:

.github/CODEOWNERS

@@ -0,0 +1 @@
+* @edigaryev @fkorotkov

README.md

@@ -1,4 +1,4 @@
-![Tart – open source virtualization for your automation needs](Resources/TartSocial.png)
+<img src="https://github.com/cirruslabs/tart/raw/main/Resources/TartSocial.png"/>
 
 *Tart* is a virtualization toolset to build, run and manage macOS and Linux virtual machines on Apple Silicon.
 Built by CI engineers for your automation needs. Here are some highlights of Tart:
@@ -22,20 +22,64 @@ Built by CI engineers for your automation needs. Here are some highlights of Tar
   </a>
 </p>
 
+Many more companies are using Tart in their internal setups. Here are a few of them:
+
+<p align="center">
+  <a href="https://ahrefs.com/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/ahrefs.png" height="65"/>
+  </a>
+  <a href="https://suran.com/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Suran.png" height="65"/>
+  </a>
+  <a href="https://symflower.com/" target=_blank>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Symflower.png" height="65"/>
+  </a>
+</p>
+
+**Note:** If your company or project is using Tart please consider [adding yourself to the list above](/Resources/Users/HowToAddYourself.md).
+
 ## Usage
 
 Try running a Tart VM on your Apple Silicon device running macOS Monterey or later (will download a 25 GB image):
 
 ```shell
 brew install cirruslabs/cli/tart
-tart clone ghcr.io/cirruslabs/macos-monterey-base:latest monterey-base
-tart run monterey-base
+tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
+tart run ventura-base
 ```
 
-![tart VM view app](Resources/TartScreenshot.png)
+<img src="https://github.com/cirruslabs/tart/raw/main/Resources/TartScreenshot.png"/>
 
 ## CI Integration
 
+Tart already powers several CI services mentioned above including our own [Cirrus CI](https://cirrus-ci.org/guide/macOS/) which offers unlimited concurrency with per-second billing.
+For services that haven't leveraged Tart yet, we offer fully managed runners via a monthly subscription.
+*Cirrus Runners* is the fastest way to get your current CI workflows to benefit from Apple Silicon hardware. No need to manage infrastructure or migrate to another CI provider.
+Please read down below about currently supported services.
+
+### Managed runners for your CI-as-a-service
+
+At the moment Cirrus Runners only supports GitHub Actions, but we are actively working on adding more options.
+Please [email us](mailto:hello@cirruslabs.org) if you are interested in a particular one.
+
+#### GitHub Actions
+
+Configuring Cirrus Runners for GitHub Actions is as simple as installing [Cirrus Runners App](https://github.com/apps/cirrus-runners).
+After successful installation and subscription configuration, use any of [Ventura images managed by us](https://github.com/cirruslabs/macos-image-templates) in `runs-on`:
+
+```yaml
+name: Test Suite
+jobs:
+  test:
+    runs-on: ghcr.io/cirruslabs/macos-ventura-xcode:latest
+```
+
+When workflows are executing you'll see Cirrus on-demand runners on your organization's settings page at `https://github.com/organizations/<ORGANIZATION>/settings/actions/runners`.
+
+<img src="https://github.com/cirruslabs/tart/raw/main/Resources/TartGHARunners.png"/>
+
+### Self-hosted CI
+
 Tart itself is only responsible for managing virtual machines, but we've built Tart support into a tool called Cirrus CLI
 also developed by Cirrus Labs. [Cirrus CLI](https://github.com/cirruslabs/cirrus-cli) is a command line tool with
 one configuration format to execute common CI steps (run a script, cache a folder, etc.) locally or in any CI system.
@@ -64,14 +108,14 @@ brew install cirruslabs/cli/cirrus
 cirrus run
 ```
 
-![Cirrus CLI Run](Resources/TartCirrusCLI.gif)
+<img src="https://github.com/cirruslabs/tart/raw/main/Resources/TartCirrusCLI.gif"/>
 
 [Cirrus CI](https://cirrus-ci.org/) already leverages Tart to power its macOS cloud infrastructure. The `.cirrus.yml`
 config from above will just work in Cirrus CI and your tasks will be executed inside Tart VMs in our cloud.
 
 **Note:** Cirrus CI only allows [images managed and regularly updated by us](https://github.com/orgs/cirruslabs/packages?tab=packages&q=macos).
 
-### Retrieving artifacts from within Tart VMs
+#### Retrieving artifacts from within Tart VMs
 
 In many cases there is a need to retrieve particular files or a folder from within a Tart virtual machine.
 For example, the below `.cirrus.yml` configuration defines a single task that builds a `tart` binary and
@@ -152,32 +196,35 @@ Please refer to `tart set --help` for additional details.
 Please refer to [Tart Packer Plugin repository](https://github.com/cirruslabs/packer-plugin-tart) for setup instructions.
 Here is an example of a template to build `monterey-base` local image based of a remote image:
 
-```json
-{
-  "builders": [
-    {
-      "name": "tart",
-      "type": "tart-cli",
-      "vm_base_name": "tartvm/vanilla:latest",
-      "vm_name": "monterey-base",
-      "cpu_count": 4,
-      "memory_gb": 8,
-      "disk_size_gb": 32,
-      "ssh_username": "admin",
-      "ssh_password": "admin",
-      "ssh_timeout": "120s"
+```hcl
+packer {
+  required_plugins {
+    tart = {
+      version = ">= 0.5.3"
+      source  = "github.com/cirruslabs/tart"
     }
-  ],
-  "provisioners": [
-    {
-      "inline": [
-        "echo 'Disabling spotlight indexing...'",
-        "sudo mdutil -a -i off"
-      ],
-      "type": "shell"
-    },
-    # more provisioners
-  ]
+  }
+}
+
+source "tart-cli" "tart" {
+  vm_base_name = "ghcr.io/cirruslabs/macos-ventura-base:latest"
+  vm_name      = "my-custom-ventura"
+  cpu_count    = 4
+  memory_gb    = 8
+  disk_size_gb = 70
+  ssh_password = "admin"
+  ssh_timeout  = "120s"
+  ssh_username = "admin"
+}
+
+build {
+  sources = ["source.tart-cli.tart"]
+
+  provisioner "shell" {
+    inline = ["echo 'Disabling spotlight indexing...'", "sudo mdutil -a -i off"]
+  }
+
+  # more provisioners
 }
 ```
 
@@ -197,6 +244,12 @@ tart login acme.io
 
 Credentials are securely stored in Keychain.
 
+In addition, Tart supports [Docker credential helpers](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers)
+if defined in `~/.docker/config.json`.
+
+Finally, `TART_REGISTRY_USERNAME` and `TART_REGISTRY_PASSWORD` environment variables allow to override authorization
+for all registries which might useful for integrating with your CI's secret management.
+
 #### Pushing a Local Image
 
 Once credentials are saved for `acme.io`, run the following command to push a local images remotely with two tags:

Resources/TartGHARunners.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23728bb5438c88b3d0826170a5fa4b7aaad0fbd94a4769c8d492c9f75b57ba81
+size 155885

Resources/TartScreenshot.png

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a43f541b1ab0b57ae2060d371cba5dbb1f5c80b89c76434b7154d8144f66e61
-size 205325
+oid sha256:7a4929ca4e02d4968904749028ada7072704d2a52cccb69335e1596452c822c7
+size 1359834

Resources/Users/Background.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d1f58f6460217434ae133c6d6e444320eb705c5dcf9efebbb215b117ae452c1
+size 589

Resources/Users/HowToAddYourself.md

@@ -0,0 +1,4 @@
+If you'd like to highlight your use of Tart, please create a `456px` by `130px` logo and create a PR
+that adds it to `README.md` in alphabetical order. Don't forget to include a small description of your usage pattern.
+
+You can refer to `Background.png` as a base for your logo.

Resources/Users/Suran.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:630c4975b9eead9e4f62e2a5375509dc0ee6cab2940ec3ffc2ae16b08d325964
+size 6765

Resources/Users/Symflower.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e48fbe90f6cc84e100fc286a5349c681cacb22358811181046d35ea298309de
+size 16379

Resources/Users/ahrefs.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c7f7f9bf0fc0c9807f3a415fe6bf773c8a416b351c5aa08de46149a13de7387
+size 5297

Sources/tart/Commands/Get.swift

@@ -0,0 +1,31 @@
+import ArgumentParser
+import Foundation
+
+struct Get: AsyncParsableCommand {
+  static var configuration = CommandConfiguration(commandName: "get", abstract: "Get a VM's configuration")
+
+  @Argument(help: "VM name")
+  var name: String
+
+  func run() async throws {
+    do {
+      let vmDir = try VMStorageLocal().open(name)
+      let vmConfig = try VMConfig(fromURL: vmDir.configURL)
+      let diskSize = try vmDir.sizeBytes() / 1000 / 1000 / 1000
+
+      print("CPU\tMemory\tDisk\tDisplay")
+
+      var s = "\(vmConfig.cpuCount)\t"
+      s += "\(vmConfig.memorySize / 1024 / 1024) MB\t"
+      s += "\(diskSize) GB\t"
+      s += "\(vmConfig.display.width)x\(vmConfig.display.height)"
+      print(s)
+
+      Foundation.exit(0)
+    } catch {
+      print(error)
+
+      Foundation.exit(1)
+    }
+  }
+}

Sources/tart/Commands/List.swift

@@ -5,9 +5,14 @@ import SwiftUI
 struct List: AsyncParsableCommand {
   static var configuration = CommandConfiguration(abstract: "List created VMs")
 
+  @Flag(name: [.short, .long], help: ArgumentHelp("Only display VM names"))
+  var quiet: Bool = false
+
   func run() async throws {
     do {
-      print("Source\tName")
+      if !quiet {
+        print("Source\tName")
+      }
 
       displayTable("local", try VMStorageLocal().list())
       displayTable("oci", try VMStorageOCI().list().map { (name, vmDir, _) in (name, vmDir) })
@@ -22,7 +27,11 @@ struct List: AsyncParsableCommand {
 
   private func displayTable(_ source: String, _ vms: [(String, VMDirectory)]) {
     for (name, _) in vms.sorted(by: { left, right in left.0 < right.0 }) {
-      print("\(source)\t\(name)")
+      if quiet {
+        print(name)
+      } else {
+        print("\(source)\t\(name)")
+      }
     }
   }
 }

Sources/tart/Commands/Run.swift

@@ -48,6 +48,20 @@ struct Run: AsyncParsableCommand {
     """, valueName: "path[:ro]"))
   var disk: [String] = []
 
+  @Option(name: [.customLong("rosetta")], help: ArgumentHelp(
+    "Attaches a Rosetta share to the guest Linux VM with a specific tag (e.g. --rosetta=\"rosetta\")",
+    discussion: """
+                Requires host to be macOS 13.0 (Ventura) with Rosetta installed. The latter can be done
+                by running "softwareupdate --install-rosetta" (without quotes) in the Terminal.app.
+
+                Note that you also have to configure Rosetta in the guest Linux VM by following the
+                steps from "Mount the Shared Directory and Register Rosetta" section here:
+                https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta#3978496
+                """,
+    valueName: "tag"
+  ))
+  var rosettaTag: String?
+
   @Option(help: ArgumentHelp("""
                              Additional directory shares with an optional read-only specifier\n(e.g. --dir=\"build:~/src/build\" --dir=\"sources:~/src/sources:ro\")
                              """, discussion: """
@@ -90,11 +104,24 @@ struct Run: AsyncParsableCommand {
   @MainActor
   func run() async throws {
     let vmDir = try VMStorageLocal().open(name)
+
+    let additionalDiskAttachments = try additionalDiskAttachments()
+
+    // Error out if the disk is locked by the host (e.g. it was mounted in Finder),
+    // see https://github.com/cirruslabs/tart/issues/323 for more details.
+    for additionalDiskAttachment in additionalDiskAttachments {
+      if try !FileLock(lockURL: additionalDiskAttachment.url).trylock() {
+        print("disk \(additionalDiskAttachment.url.path) seems to be already in use, "
+          + "unmount it first in Finder")
+        Foundation.exit(1)
+      }
+    }
+
     vm = try VM(
       vmDir: vmDir,
       network: userSpecifiedNetwork(vmDir: vmDir) ?? NetworkShared(),
-      additionalDiskAttachments: additionalDiskAttachments(),
-      directoryShares: directoryShares()
+      additionalDiskAttachments: additionalDiskAttachments,
+      directorySharingDevices: directoryShares() + rosettaDirectoryShare()
     )
 
     let vncImpl: VNC? = try {
@@ -108,6 +135,24 @@ struct Run: AsyncParsableCommand {
       }
     }()
 
+    // Lock the VM
+    //
+    // More specifically, lock the "config.json", because we can't lock
+    // directories with fcntl(2)-based locking and we better not interfere
+    // with the VM's disk and NVRAM, because they are opened (and even seem
+    // to be locked) directly by the Virtualization.Framework's process.
+    //
+    // Note that due to "completely stupid semantics"[1] of the fcntl-based
+    // file locking, we need to acquire the lock after we read the VM's
+    // configuration file, otherwise we will loose the lock.
+    //
+    // [1]: https://man.openbsd.org/fcntl
+    let lock = try PIDLock(lockURL: vmDir.configURL)
+    if try !lock.trylock() {
+      print("Virtual machine \"\(name)\" is already running!")
+      Foundation.exit(2)
+    }
+
     let task = Task {
       do {
         if let vncImpl = vncImpl {
@@ -129,11 +174,6 @@ struct Run: AsyncParsableCommand {
 
         Foundation.exit(0)
       } catch {
-        if error.localizedDescription.contains("Failed to lock auxiliary storage.") {
-          print("Virtual machine \"\(name)\" is already running!")
-          Foundation.exit(2)
-        }
-
         print(error)
         Foundation.exit(1)
       }
@@ -216,8 +256,22 @@ struct Run: AsyncParsableCommand {
     return result
   }
 
-  func directoryShares() throws -> [DirectoryShare] {
-    var result: [DirectoryShare] = []
+  func directoryShares() throws -> [VZDirectorySharingDeviceConfiguration] {
+    if dir.isEmpty {
+      return []
+    }
+
+    guard #available(macOS 13, *) else {
+      throw UnsupportedOSError("directory sharing", "is")
+    }
+
+    struct DirectoryShare {
+      let name: String
+      let path: URL
+      let readOnly: Bool
+    }
+
+    var directoryShares: [DirectoryShare] = []
 
     for rawDir in dir {
       let splits = rawDir.split(maxSplits: 2) { $0 == ":" }
@@ -238,10 +292,46 @@ struct Run: AsyncParsableCommand {
 
       let (name, path) = (String(splits[0]), String(splits[1]))
 
-      result.append(DirectoryShare(name: name, path: URL(fileURLWithPath: NSString(string: path).expandingTildeInPath), readOnly: readOnly))
+      directoryShares.append(DirectoryShare(
+        name: name,
+        path: URL(fileURLWithPath: NSString(string: path).expandingTildeInPath),
+        readOnly: readOnly)
+      )
     }
 
-    return result
+    var directories: [String : VZSharedDirectory] = Dictionary()
+    directoryShares.forEach { directories[$0.name] = VZSharedDirectory(url: $0.path, readOnly: $0.readOnly) }
+
+    let automountTag = VZVirtioFileSystemDeviceConfiguration.macOSGuestAutomountTag
+    let sharingDevice = VZVirtioFileSystemDeviceConfiguration(tag: automountTag)
+    sharingDevice.share = VZMultipleDirectoryShare(directories: directories)
+
+    return [sharingDevice]
+  }
+
+  private func rosettaDirectoryShare() throws -> [VZDirectorySharingDeviceConfiguration] {
+    guard let rosettaTag = rosettaTag else {
+      return []
+    }
+
+    guard #available(macOS 13, *) else {
+      throw UnsupportedOSError("Rosetta directory share", "is")
+    }
+
+    switch VZLinuxRosettaDirectoryShare.availability {
+    case .notInstalled:
+      throw UnsupportedOSError("Rosetta directory share", "is", "that have Rosetta installed")
+    case .notSupported:
+      throw UnsupportedOSError("Rosetta directory share", "is", "running Apple silicon")
+    default:
+      break
+    }
+
+    try VZVirtioFileSystemDeviceConfiguration.validateTag(rosettaTag)
+    let device = VZVirtioFileSystemDeviceConfiguration(tag: rosettaTag)
+    device.share = try VZLinuxRosettaDirectoryShare()
+
+    return [device]
   }
 
   private func runUI() {

Sources/tart/Commands/Set.swift

@@ -11,7 +11,7 @@ struct Set: AsyncParsableCommand {
   var cpu: UInt16?
 
   @Option(help: "VM memory size in megabytes")
-  var memory: UInt16?
+  var memory: UInt64?
 
   @Option(help: "VM display resolution in a format of <width>x<height>. For example, 1200x800")
   var display: VMDisplayConfig?
@@ -29,7 +29,7 @@ struct Set: AsyncParsableCommand {
       }
 
       if let memory = memory {
-        try vmConfig.setMemory(memorySize: UInt64(memory) * 1024 * 1024)
+        try vmConfig.setMemory(memorySize: memory * 1024 * 1024)
       }
 
       if let display = display {

Sources/tart/Commands/Stop.swift

@@ -0,0 +1,72 @@
+import ArgumentParser
+import Foundation
+import System
+import SwiftDate
+
+struct Stop: AsyncParsableCommand {
+  static var configuration = CommandConfiguration(commandName: "stop", abstract: "Stop a VM")
+
+  @Argument(help: "VM name")
+  var name: String
+
+  @Option(name: [.short, .long], help: "Seconds to wait for graceful termination before forcefully terminating the VM")
+  var timeout: UInt64 = 30
+
+  func run() async throws {
+    do {
+      let vmDir = try VMStorageLocal().open(name)
+      let lock = try PIDLock(lockURL: vmDir.configURL)
+
+      // Find the VM's PID
+      var pid = try lock.pid()
+      if pid == 0 {
+        print("VM \(name) is not running")
+
+        Foundation.exit(2)
+      }
+
+      // Try to gracefully terminate the VM
+      //
+      // Note that we don't check the return code here
+      // to provide a clean exit from "tart stop" in cases
+      // when the VM is already shutting down and we hit
+      // a race condition.
+      //
+      // We check the return code in the kill(2) below, though,
+      // because it's a less common scenario and it would be
+      // nice to know for the user that we've tried all methods
+      // and failed to shutdown the VM.
+      kill(pid, SIGINT)
+
+      // Ensure that the VM has terminated
+      var gracefulWaitDuration = Measurement(value: Double(timeout), unit: UnitDuration.seconds)
+      let gracefulTickDuration = Measurement(value: Double(100), unit: UnitDuration.milliseconds)
+
+      while gracefulWaitDuration.value > 0 {
+        pid = try lock.pid()
+        if pid == 0 {
+          Foundation.exit(0)
+        }
+
+        try await Task.sleep(nanoseconds: UInt64(gracefulTickDuration.converted(to: .nanoseconds).value))
+        gracefulWaitDuration = gracefulWaitDuration - gracefulTickDuration
+      }
+
+      // Seems that VM is still running, proceed with forceful termination
+      let ret = kill(pid, SIGKILL)
+      if ret != 0 {
+        let details = Errno(rawValue: CInt(errno))
+
+        print("failed to forcefully terminate the VM \(name): \(details)")
+
+        Foundation.exit(1)
+      }
+
+      Foundation.exit(0)
+    } catch {
+      print(error)
+
+      Foundation.exit(1)
+    }
+  }
+}

Sources/tart/Credentials/EnvironmentCredentialsProvider.swift

@@ -0,0 +1,15 @@
+import Foundation
+
+class EnvironmentCredentialsProvider: CredentialsProvider {
+  func retrieve(host: String) throws -> (String, String)? {
+    let username = ProcessInfo.processInfo.environment["TART_REGISTRY_USERNAME"]
+    let password = ProcessInfo.processInfo.environment["TART_REGISTRY_PASSWORD"]
+    if let username = username, let password = password {
+      return (username, password)
+    }
+    return nil
+  }
+
+  func store(host: String, user: String, password: String) throws {
+  }
+}

Sources/tart/Fetcher.swift

@@ -1,46 +1,67 @@
 import Foundation
 import AsyncAlgorithms
 
-class Fetcher: NSObject, URLSessionTaskDelegate, URLSessionDelegate, URLSessionDataDelegate {
-    let responseCh = AsyncThrowingChannel<URLResponse, Error>()
+fileprivate let urlSession = createURLSession()
+
+class Fetcher {
+  static func fetch(_ request: URLRequest, viaFile: Bool = false) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
+    if viaFile {
+      return try await fetchViaFile(request)
+    }
+
+    return try await fetchViaMemory(request)
+  }
+
+  private static func fetchViaMemory(_ request: URLRequest) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
     let dataCh = AsyncThrowingChannel<Data, Error>()
 
-    func fetch(_ request: URLRequest) async throws -> (AsyncThrowingChannel<Data, Error>, URLResponse) {
-        let task = URLSession.shared.dataTask(with: request)
-        task.delegate = self
-        task.resume()
+    let (data, response) = try await urlSession.data(for: request)
 
-        // Wait for the response and only then return
-        var iter = responseCh.makeAsyncIterator()
-        let response = try await iter.next()!
+    Task {
+      await dataCh.send(data)
 
-        return (dataCh, response)
+      dataCh.finish()
     }
 
-    func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive response: URLResponse) async -> URLSession.ResponseDisposition {
-        await responseCh.send(response)
+    return (dataCh, response as! HTTPURLResponse)
+  }
 
-        return .allow
+  private static func fetchViaFile(_ request: URLRequest) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
+    let dataCh = AsyncThrowingChannel<Data, Error>()
+
+    let (fileURL, response) = try await urlSession.download(for: request)
+
+    // Acquire a handle to the downloaded file and then remove it.
+    //
+    // This keeps a working reference to that file, yet we don't
+    // have to deal with the cleanup any more.
+    let fh = try FileHandle(forReadingFrom: fileURL)
+    try FileManager.default.removeItem(at: fileURL)
+
+    Task {
+      while let data = try fh.read(upToCount: 64 * 1024 * 1024) {
+        await dataCh.send(data)
+      }
+
+      dataCh.finish()
     }
 
-    func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {
-        let sema = DispatchSemaphore(value: 0)
+    return (dataCh, response as! HTTPURLResponse)
+  }
+}
 
-        Task {
-            await dataCh.send(data)
-            sema.signal()
-        }
+fileprivate func createURLSession() -> URLSession {
+  let config = URLSessionConfiguration.default
 
-        sema.wait()
-    }
+  // Harbor expects a CSRF token to be present if the HTTP client
+  // carries a session cookie between its requests[1] and fails if
+  // it was not present[2].
+  //
+  // To fix that, we disable the automatic cookies carry in URLSession.
+  //
+  // [1]: https://github.com/goharbor/harbor/blob/a4c577f9ec4f18396207a5e686433a6ba203d4ef/src/server/middleware/csrf/csrf.go#L78
+  // [2]: https://github.com/cirruslabs/tart/issues/295
+  config.httpShouldSetCookies = false
 
-    func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
-        if let error = error {
-            // Premature termination
-            responseCh.fail(error)
-            dataCh.fail(error)
-        } else {
-            dataCh.finish()
-        }
-    }
+  return URLSession(configuration: config)
 }

Sources/tart/OCI/Registry.swift

@@ -100,7 +100,7 @@ class Registry {
 
   init(urlComponents: URLComponents,
        namespace: String,
-       credentialsProviders: [CredentialsProvider] = [DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
+       credentialsProviders: [CredentialsProvider] = [EnvironmentCredentialsProvider(), DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
   ) throws {
     baseURL = urlComponents.url!
     self.namespace = namespace
@@ -111,7 +111,7 @@ class Registry {
     host: String,
     namespace: String,
     insecure: Bool = false,
-    credentialsProviders: [CredentialsProvider] = [DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
+    credentialsProviders: [CredentialsProvider] = [EnvironmentCredentialsProvider(), DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
   ) throws {
     let proto = insecure ? "http" : "https"
     let baseURLComponents = URLComponents(string: proto + "://" + host + "/v2/")!
@@ -226,7 +226,7 @@ class Registry {
   }
 
   public func pullBlob(_ digest: String, handler: (Data) throws -> Void) async throws {
-    let (channel, response) = try await channelRequest(.GET, endpointURL("\(namespace)/blobs/\(digest)"))
+    let (channel, response) = try await channelRequest(.GET, endpointURL("\(namespace)/blobs/\(digest)"), viaFile: true)
     if response.statusCode != HTTPCode.Ok.rawValue {
       let body = try await channel.asData().asText()
       throw RegistryError.UnexpectedHTTPStatusCode(when: "pulling blob", code: response.statusCode,
@@ -266,7 +266,8 @@ class Registry {
     headers: Dictionary<String, String> = Dictionary(),
     parameters: Dictionary<String, String> = Dictionary(),
     body: Data? = nil,
-    doAuth: Bool = true
+    doAuth: Bool = true,
+    viaFile: Bool = false
   ) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
     var urlComponents = urlComponents
 
@@ -292,12 +293,12 @@ class Registry {
       currentAuthToken = nil
     }
 
-    var (channel, response) = try await authAwareRequest(request: request)
+    var (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile)
 
     if doAuth && response.statusCode == HTTPCode.Unauthorized.rawValue {
       _ = try await channel.asData()
       try await auth(response: response)
-      (channel, response) = try await authAwareRequest(request: request)
+      (channel, response) = try await authAwareRequest(request: request, viaFile: viaFile)
     }
 
     return (channel, response)
@@ -372,7 +373,7 @@ class Registry {
     return nil
   }
 
-  private func authAwareRequest(request: URLRequest) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
+  private func authAwareRequest(request: URLRequest, viaFile: Bool = false) async throws -> (AsyncThrowingChannel<Data, Error>, HTTPURLResponse) {
     var request = request
 
     if let token = currentAuthToken {
@@ -380,8 +381,6 @@ class Registry {
       request.addValue(value, forHTTPHeaderField: name)
     }
 
-    let (channel, response) = try await Fetcher().fetch(request)
-
-    return (channel, response as! HTTPURLResponse)
+    return try await Fetcher.fetch(request, viaFile: viaFile)
   }
 }

Sources/tart/PIDLock.swift

@@ -0,0 +1,52 @@
+import Foundation
+import System
+
+class PIDLock {
+  let url: URL
+  let fd: Int32
+
+  init(lockURL: URL) throws {
+    url = lockURL
+    fd = open(lockURL.path, O_RDWR)
+  }
+
+  deinit {
+    close(fd)
+  }
+
+  func trylock() throws -> Bool {
+    let (locked, _) = try lockWrapper(F_SETLK, F_WRLCK, "failed to lock \(url)")
+    return locked
+  }
+
+  func lock() throws {
+    _ = try lockWrapper(F_SETLKW, F_WRLCK, "failed to lock \(url)")
+  }
+
+  func unlock() throws {
+    _ = try lockWrapper(F_SETLK, F_UNLCK, "failed to unlock \(url)")
+  }
+
+  func pid() throws -> pid_t {
+    let (_, result) = try lockWrapper(F_GETLK, F_RDLCK, "failed to get lock \(url) status")
+
+    return result.l_pid
+  }
+
+  func lockWrapper(_ operation: Int32, _ type: Int32, _ message: String) throws -> (Bool, flock) {
+    var result = flock(l_start: 0, l_len: 0, l_pid: 0, l_type: Int16(type), l_whence: Int16(SEEK_SET))
+
+    let ret = fcntl(fd, operation, &result)
+    if ret != 0 {
+      if operation == F_SETLK && errno == EAGAIN {
+        return (false, result)
+      }
+
+      let details = Errno(rawValue: CInt(errno))
+
+      throw RuntimeError("\(message): \(details)")
+    }
+
+    return (true, result)
+  }
+}

Sources/tart/Passphrase/Words.swift

@@ -24,7 +24,7 @@ let passphrases = [
   "act",
   "action",
   "actor",
-  "    actress",
+  "actress",
   "actual",
   "adapt",
   "add",
@@ -829,7 +829,7 @@ let passphrases = [
   "grow",
   "grunt",
   "guard",
-  "    guess",
+  "guess",
   "guide",
   "guilt",
   "guitar",
@@ -1476,7 +1476,7 @@ let passphrases = [
   "retire",
   "retreat",
   "return",
-  "    reunion",
+  "reunion",
   "reveal",
   "review",
   "reward",
@@ -1764,7 +1764,7 @@ let passphrases = [
   "swim",
   "swing",
   "switch",
-  "    sword",
+  "sword",
   "symbol",
   "symptom",
   "syrup",

Sources/tart/Root.swift

@@ -20,6 +20,7 @@ struct Root: AsyncParsableCommand {
       Clone.self,
       Run.self,
       Set.self,
+      Get.self,
       List.self,
       Login.self,
       IP.self,
@@ -27,6 +28,7 @@ struct Root: AsyncParsableCommand {
       Push.self,
       Prune.self,
       Rename.self,
+      Stop.self,
       Delete.self,
     ])
 

Sources/tart/VM.swift

@@ -14,8 +14,8 @@ struct DownloadFailed: Error {
 struct UnsupportedOSError: Error, CustomStringConvertible {
   let description: String
 
-  init(_ what: String, _ plural: String) {
-    description = "error: \(what) \(plural) only supported on hosts running macOS 13.0 (Ventura) or newer"
+  init(_ what: String, _ plural: String, _ requires: String = "running macOS 13.0 (Ventura) or newer") {
+    description = "error: \(what) \(plural) only supported on hosts \(requires)"
   }
 }
 
@@ -40,7 +40,7 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
   init(vmDir: VMDirectory,
        network: Network = NetworkShared(),
        additionalDiskAttachments: [VZDiskImageStorageDeviceAttachment] = [],
-       directoryShares: [DirectoryShare] = []
+       directorySharingDevices: [VZDirectorySharingDeviceConfiguration] = []
   ) throws {
     name = vmDir.name
     config = try VMConfig.init(fromURL: vmDir.configURL)
@@ -54,7 +54,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     let configuration = try Self.craftConfiguration(diskURL: vmDir.diskURL,
       nvramURL: vmDir.nvramURL, vmConfig: config,
       network: network, additionalDiskAttachments: additionalDiskAttachments,
-            directoryShares: directoryShares)
+      directorySharingDevices: directorySharingDevices
+    )
     virtualMachine = VZVirtualMachine(configuration: configuration)
 
     super.init()
@@ -63,9 +64,9 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
 
   static func retrieveIPSW(remoteURL: URL) async throws -> URL {
     // Check if we already have this IPSW in cache
-    let (channel, response) = try await Fetcher().fetch(URLRequest(url: remoteURL))
+    let (channel, response) = try await Fetcher.fetch(URLRequest(url: remoteURL), viaFile: true)
 
-    if let hash = (response as! HTTPURLResponse).value(forHTTPHeaderField: "x-amz-meta-digest-sha256") {
+    if let hash = response.value(forHTTPHeaderField: "x-amz-meta-digest-sha256") {
       let ipswLocation = try IPSWCache().locationFor(fileName: "sha256:\(hash).ipsw")
 
       if FileManager.default.fileExists(atPath: ipswLocation.path) {
@@ -130,7 +131,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     ipswURL: URL,
     diskSizeGB: UInt16,
     network: Network = NetworkShared(),
-    additionalDiskAttachments: [VZDiskImageStorageDeviceAttachment] = []
+    additionalDiskAttachments: [VZDiskImageStorageDeviceAttachment] = [],
+    directorySharingDevices: [VZDirectorySharingDeviceConfiguration] = []
   ) async throws {
     var ipswURL = ipswURL
 
@@ -138,6 +140,11 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
       ipswURL = try await VM.retrieveIPSW(remoteURL: ipswURL)
     }
 
+    // We create a temporary TART_HOME directory in tests, which has its "cache" folder symlinked
+    // to the users Tart cache directory (~/.tart/cache). However, the Virtualization.Framework
+    // cannot deal with paths that contain symlinks, so expand them here first.
+    ipswURL.resolveSymlinksInPath()
+
     // Load the restore image and try to get the requirements
     // that match both the image and our platform
     let image = try await withCheckedThrowingContinuation { continuation in
@@ -172,7 +179,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     let configuration = try Self.craftConfiguration(diskURL: vmDir.diskURL, nvramURL: vmDir.nvramURL,
       vmConfig: config, network: network,
       additionalDiskAttachments: additionalDiskAttachments,
-      directoryShares: [])
+      directorySharingDevices: directorySharingDevices
+    )
     virtualMachine = VZVirtualMachine(configuration: configuration)
 
     super.init()
@@ -248,7 +256,7 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     vmConfig: VMConfig,
     network: Network = NetworkShared(),
     additionalDiskAttachments: [VZDiskImageStorageDeviceAttachment],
-    directoryShares: [DirectoryShare]
+    directorySharingDevices: [VZDirectorySharingDeviceConfiguration]
   ) throws -> VZVirtualMachineConfiguration {
     let configuration = VZVirtualMachineConfiguration()
 
@@ -292,19 +300,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     // Entropy
     configuration.entropyDevices = [VZVirtioEntropyDeviceConfiguration()]
 
-    // Directory share
-    if #available(macOS 13, *) {
-      var directories: [String : VZSharedDirectory] = Dictionary()
-      directoryShares.forEach { directories[$0.name] = VZSharedDirectory(url: $0.path, readOnly: $0.readOnly) }
-
-      let automountTag = VZVirtioFileSystemDeviceConfiguration.macOSGuestAutomountTag
-      let sharingDevice = VZVirtioFileSystemDeviceConfiguration(tag: automountTag)
-      sharingDevice.share = VZMultipleDirectoryShare(directories: directories)
-
-      configuration.directorySharingDevices = [sharingDevice]
-    } else if !directoryShares.isEmpty {
-      throw UnsupportedOSError("directory sharing", "is")
-    }
+    // Directory sharing devices
+    configuration.directorySharingDevices = directorySharingDevices
 
     try configuration.validate()
 
@@ -317,7 +314,7 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
   }
 
   func virtualMachine(_ virtualMachine: VZVirtualMachine, didStopWithError error: Error) {
-    print("guest has stopped the virtual machine due to error")
+    print("guest has stopped the virtual machine due to error: \(error)")
     sema.signal()
   }
 
@@ -326,9 +323,3 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     sema.signal()
   }
 }
-
-struct DirectoryShare {
-  let name: String
-  let path: URL
-  let readOnly: Bool
-}

Sources/tart/VMDirectory.swift

@@ -1,12 +1,6 @@
 import Foundation
 import Virtualization
 
-struct UninitializedVMDirectoryError: Error {
-}
-
-struct AlreadyInitializedVMDirectoryError: Error {
-}
-
 struct VMDirectory: Prunable {
   var baseURL: URL
 
@@ -47,7 +41,7 @@ struct VMDirectory: Prunable {
 
   func initialize(overwrite: Bool = false) throws {
     if !overwrite && initialized {
-      throw AlreadyInitializedVMDirectoryError()
+      throw RuntimeError("VM directory is already initialized, preventing overwrite")
     }
 
     try FileManager.default.createDirectory(at: baseURL, withIntermediateDirectories: true, attributes: nil)
@@ -58,8 +52,13 @@ struct VMDirectory: Prunable {
   }
 
   func validate() throws {
+    if !FileManager.default.fileExists(atPath: baseURL.path) {
+      throw RuntimeError("the specified VM does not exist")
+    }
+
     if !initialized {
-      throw UninitializedVMDirectoryError()
+      throw RuntimeError("VM is missing some of its files (\(configURL.lastPathComponent),"
+        + " \(diskURL.lastPathComponent) or \(nvramURL.lastPathComponent))")
     }
   }
 

integration-tests/conftest.py

@@ -0,0 +1,16 @@
+import pytest
+
+from tart import Tart
+from docker_registry import DockerRegistry
+
+
+@pytest.fixture(scope="class")
+def tart():
+    with Tart() as tart:
+        yield tart
+
+
+@pytest.fixture(scope="class")
+def docker_registry():
+    with DockerRegistry() as docker_registry:
+        yield docker_registry

integration-tests/docker_registry.py

@@ -0,0 +1,20 @@
+import requests
+
+from testcontainers.core.waiting_utils import wait_container_is_ready
+from testcontainers.general import DockerContainer
+
+
+class DockerRegistry(DockerContainer):
+    _default_exposed_port = 5000
+
+    def __init__(self):
+        super().__init__("registry:2")
+        self.with_exposed_ports(self._default_exposed_port)
+
+    @wait_container_is_ready(requests.exceptions.ConnectionError)
+    def remote_name(self, for_vm: str):
+        exposed_port = self.get_exposed_port(self._default_exposed_port)
+
+        requests.get(f"http://127.0.0.1:{exposed_port}/v2/")
+
+        return f"127.0.0.1:{exposed_port}/tart/{for_vm}:latest"

integration-tests/requirements.txt

@@ -0,0 +1,5 @@
+pytest
+testcontainers
+requests
+bitmath
+pytest-dependency

integration-tests/tart.py

@@ -0,0 +1,33 @@
+import tempfile
+import os
+import subprocess
+
+
+class Tart:
+    def __init__(self):
+        self.tmp_dir = tempfile.TemporaryDirectory(dir=os.environ.get("CIRRUS_WORKING_DIR"))
+
+        # Link to the users IPSW cache to make things faster
+        src = os.path.join(os.path.expanduser("~"), ".tart", "cache", "IPSWs")
+        dst = os.path.join(self.tmp_dir.name, "cache", "IPSWs")
+        os.makedirs(os.path.join(self.tmp_dir.name, "cache"))
+        os.symlink(src, dst)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.tmp_dir.cleanup()
+
+    def home(self) -> str:
+        return self.tmp_dir.name
+
+    def run(self, args):
+        env = os.environ.copy()
+        env.update({"TART_HOME": self.tmp_dir.name})
+
+        completed_process = subprocess.run(["tart"] + args, env=env, capture_output=True)
+
+        completed_process.check_returncode()
+
+        return completed_process.stdout.decode("utf-8"), completed_process.stderr.decode("utf-8")

integration-tests/test_clone.py

@@ -0,0 +1,10 @@
+def test_clone(tart):
+    # Create a Linux VM (because we can create it really fast)
+    tart.run(["create", "--linux", "debian"])
+
+    # Clone the VM
+    tart.run(["clone", "debian", "ubuntu"])
+
+    # Ensure that we have now 2 VMs
+    stdout, _, = tart.run(["list", "--quiet"])
+    assert stdout == "debian\nubuntu\n"

integration-tests/test_create.py

@@ -0,0 +1,16 @@
+def test_create_macos(tart):
+    # Create a macOS VM
+    tart.run(["create", "--from-ipsw", "latest", "macos-vm"])
+
+    # Ensure that the VM was created
+    stdout, _ = tart.run(["list", "--quiet"])
+    assert stdout == "macos-vm\n"
+
+
+def test_create_linux(tart):
+    # Create a Linux VM
+    tart.run(["create", "--linux", "linux-vm"])
+
+    # Ensure that the VM was created
+    stdout, _ = tart.run(["list", "--quiet"])
+    assert stdout == "linux-vm\n"

integration-tests/test_delete.py

@@ -0,0 +1,14 @@
+def test_delete(tart):
+    # Create a Linux VM (because we can create it really fast)
+    tart.run(["create", "--linux", "debian"])
+
+    # Ensure that the VM exists
+    stdout, _, = tart.run(["list", "--quiet"])
+    assert stdout == "debian\n"
+
+    # Delete the VM
+    tart.run(["delete", "debian"])
+
+    # Ensure that the VM was removed
+    stdout, _, = tart.run(["list", "--quiet"])
+    assert stdout == ""

integration-tests/test_oci.py

@@ -0,0 +1,55 @@
+import os
+import tempfile
+import timeit
+import uuid
+
+import bitmath
+import pytest
+
+amount_to_transfer = bitmath.GB(1)
+minimal_speed_per_second = bitmath.Mb(100)
+
+
+class TestOCI:
+    @pytest.mark.dependency()
+    def test_push_speed(self, tart, vm_with_random_disk, docker_registry):
+        start = timeit.default_timer()
+        tart.run(["push", "--insecure", vm_with_random_disk, docker_registry.remote_name(vm_with_random_disk)])
+        stop = timeit.default_timer()
+
+        actual_speed_per_second = self._calculate_speed_per_second(amount_to_transfer, stop - start)
+        assert actual_speed_per_second > minimal_speed_per_second
+
+    @pytest.mark.dependency(depends=["TestOCI::test_push_speed"])
+    def test_pull_speed(self, tart, vm_with_random_disk, docker_registry):
+        start = timeit.default_timer()
+        tart.run(["pull", "--insecure", docker_registry.remote_name(vm_with_random_disk)])
+        stop = timeit.default_timer()
+
+        actual_speed_per_second = self._calculate_speed_per_second(amount_to_transfer, stop - start)
+        assert actual_speed_per_second > minimal_speed_per_second
+
+    @staticmethod
+    def _calculate_speed_per_second(amount_transferred, time_taken):
+        return (amount_transferred / time_taken).best_prefix(bitmath.SI)
+
+
+@pytest.fixture(scope="class")
+def vm_with_random_disk(tart):
+    vm_name = str(uuid.uuid4())
+
+    # Create a VM (Linux for speed's sake)
+    tart.run(["create", "--linux", vm_name])
+
+    # Populate VM's disk with "amount_to_transfer" of random bytes
+    # to effectively disable Tart's OCI blob compression
+    disk_path = os.path.join(tart.home(), "vms", vm_name, "disk.img")
+
+    with tempfile.NamedTemporaryFile(delete=False) as tf:
+        tf.write(os.urandom(amount_to_transfer.bytes))
+        tf.close()
+        os.rename(tf.name, disk_path)
+
+    yield vm_name
+
+    tart.run(["delete", vm_name])

integration-tests/test_rename.py

@@ -0,0 +1,10 @@
+def test_rename(tart):
+    # Create a Linux VM (because we can create it really fast)
+    tart.run(["create", "--linux", "debian"])
+
+    # Rename that VM
+    tart.run(["rename", "debian", "ubuntu"])
+
+    # Ensure that the VM is now named "ubuntu"
+    stdout, _, = tart.run(["list", "--quiet"])
+    assert stdout == "ubuntu\n"