Optimize DiskV2 Deduplication (#878)

* Revert "Lowercase `tart.app` (#751)"

This reverts commit a9e2a19015.

* Optimize DiskV2 deduplication logic

In case we cloned `disk.img` from a local image, check if data at offset has the expected contents already.

* Hole punch only if needed

* Calculate hash only if needed

* subdataChunks optimization

* Reapply "Lowercase `tart.app` (#751)"

This reverts commit e74e9c845a.

* format

* Save at least 1GB on deduplication logic

* Build separately

* Revert "subdataChunks optimization"

This reverts commit e59382aeba.

* Another optimization

* Removed debug log

* reformat

* Revert "Hole punch only if needed"

This reverts commit 8c569fc5

.cirrus.yml

@@ -8,11 +8,11 @@ task:
       name: dev-mini
     resources:
       tart-vms: 1
+  build_script:
+    - swift build
   test_script:
     - swift test
   integration_test_script:
-    # Build Tart
-    - swift build
     - codesign --sign - --entitlements Resources/tart-dev.entitlements --force .build/debug/tart
     - export PATH=$(pwd)/.build/arm64-apple-macosx/debug:$PATH
     # Run integration tests

Package.resolved

@@ -1,5 +1,5 @@
 {
-  "originHash" : "6d48639bc0ea02002de0b4f38fe3fce0ddc9d174f2e56180c2ffcbedb7391ef8",
+  "originHash" : "2c514a4a1d7e106713db744bee89edb40d75da63e6611990ec2f4b0da53c0455",
   "pins" : [
     {
       "identity" : "antlr4",
@@ -118,6 +118,15 @@
         "version" : "1.8.0"
       }
     },
+    {
+      "identity" : "swift-xattr",
+      "kind" : "remoteSourceControl",
+      "location" : "https://github.com/jozefizso/swift-xattr",
+      "state" : {
+        "revision" : "f8605af7b3290dbb235fb182ec6e9035d0c8c3ac",
+        "version" : "3.0.0"
+      }
+    },
     {
       "identity" : "swiftdate",
       "kind" : "remoteSourceControl",

Package.swift

@@ -24,6 +24,7 @@ let package = Package(
     .package(url: "https://github.com/orchetect/SwiftRadix", from: "1.3.1"),
     .package(url: "https://github.com/groue/Semaphore", from: "0.0.8"),
     .package(url: "https://github.com/fumoboy007/swift-retry", from: "0.2.3"),
+    .package(url: "https://github.com/jozefizso/swift-xattr", from: "3.0.0"),
   ],
   targets: [
     .executableTarget(name: "tart", dependencies: [
@@ -40,6 +41,7 @@ let package = Package(
       .product(name: "SwiftRadix", package: "SwiftRadix"),
       .product(name: "Semaphore", package: "Semaphore"),
       .product(name: "DMRetry", package: "swift-retry"),
+      .product(name: "XAttr", package: "swift-xattr"),
     ], exclude: [
       "OCI/Reference/Makefile",
       "OCI/Reference/Reference.g4",

README.md

@@ -17,23 +17,23 @@ service — a drop-in replacement for the standard GitHub-hosted runners, offeri
   </a>
 </p>
 
-Many companies are using Tart in their internal setups. Here are a few of them:
+Many companies are using Tart in their internal setups. Here are just a few of them:
 
 <p align="center">
   <a href="https://atlassian.com/" target=_blank>
     <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Atlassian.png" height="65"/>
   </a>
-  <a href="https://krisp.ai/" target=_blank>
+  <a href="https://www.figma.com/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Krisp.png" height="65"/>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Figma.png" height="65"/>
   </a>
   <a href="https://mullvad.net/" target=_blank>
     <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Mullvad.png" height="65"/>
   </a>
-  <a href="https://shape.dk/" target=_blank>
+  <a href="https://krisp.ai/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/shape.png" height="65"/>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Krisp.png" height="65"/>
   </a>
-  <a href="https://suran.com/" target=_blank>
+  <a href="https://testingbot.com/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Suran.png" height="65"/>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/TestingBot.png" height="65"/>
   </a>
   <a href="https://symflower.com/" target=_blank>
     <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Symflower.png" height="65"/>
@@ -41,18 +41,18 @@ Many companies are using Tart in their internal setups. Here are a few of them:
   <a href="https://transloadit.com/" target=_blank>
     <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Transloadit.png" height="65"/>
   </a>
-  <a href="https://uphold.com/" target=_blank>
+  <a href="https://cirrus-ci.org/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Uphold.png" height="65"/>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/CirrusCI.png" height="65"/>
   </a>
   <a href="https://www.pitsdatarecovery.net/" target=_blank>
     <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/PITSGlobalDataRecoveryServices.png" height="65"/>
   </a>
-  <a href="https://ahrefs.com/" target=_blank>
+  <a href="https://expo.dev/" target=_blank>
-    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/ahrefs.png" height="65"/>
+    <img src="https://github.com/cirruslabs/tart/raw/main/Resources/Users/Expo.png" height="65"/>
   </a>
 </p>
 
-**Note:** If your company or project is using Tart please consider [adding yourself to the list above](/Resources/Users/HowToAddYourself.md).
+**Note:** If your company or project is using Tart please consider [sharing with the community](https://github.com/cirruslabs/tart/discussions/857).
 
 <p align="center">
   <a href="https://aws.amazon.com/marketplace/pp/prodview-qczco34wlkdws?utm_source=github&utm_medium=referral" target=_blank>

Resources/Users/HowToAddYourself.md

@@ -1,4 +0,0 @@
-If you'd like to highlight your use of Tart, please create a `456px` by `130px` logo and create a PR
-that adds it to `README.md` in alphabetical order. Don't forget to include a small description of your usage pattern.
-
-You can refer to `Background.png` as a base for your logo.

Sources/tart/Commands/List.swift

@@ -7,6 +7,7 @@ fileprivate struct VMInfo: Encodable {
   let Name: String
   let Disk: Int
   let Size: Int
+  let SizeOnDisk: Int
   let Running: Bool
   let State: String
 }
@@ -38,13 +39,13 @@ struct List: AsyncParsableCommand {
 
     if source == nil || source == "local" {
       infos += sortedInfos(try VMStorageLocal().list().map { (name, vmDir) in
-        try VMInfo(Source: "local", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
+        try VMInfo(Source: "local", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), SizeOnDisk: vmDir.allocatedSizeGB() - vmDir.deduplicatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
       })
     }
 
     if source == nil || source == "oci" {
       infos += sortedInfos(try VMStorageOCI().list().map { (name, vmDir, _) in
-        try VMInfo(Source: "OCI", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
+        try VMInfo(Source: "OCI", Name: name, Disk: vmDir.sizeGB(), Size: vmDir.allocatedSizeGB(), SizeOnDisk: vmDir.allocatedSizeGB() - vmDir.deduplicatedSizeGB(), Running: vmDir.running(), State: vmDir.state().rawValue)
       })
     }
 

Sources/tart/Commands/Login.swift

@@ -49,9 +49,10 @@ struct Login: AsyncParsableCommand {
     ])
 
     if !noValidate {
+      let registry = try Registry(host: host, namespace: "", insecure: insecure,
+                                  credentialsProviders: [credentialsProvider])
+
       do {
-        let registry = try Registry(host: host, namespace: "", insecure: insecure,
-                                    credentialsProviders: [credentialsProvider])
         try await registry.ping()
       } catch {
         throw RuntimeError.InvalidCredentials("invalid credentials: \(error)")

Sources/tart/Commands/Push.swift

@@ -15,6 +15,9 @@ struct Push: AsyncParsableCommand {
   @Flag(help: "connect to the OCI registry via insecure HTTP protocol")
   var insecure: Bool = false
 
+  @Option(help: "network concurrency to use when pushing a local VM to the OCI-compatible registry")
+  var concurrency: UInt = 4
+
   @Option(help: ArgumentHelp("chunk size in MB if registry supports chunked uploads",
                              discussion: """
                              By default monolithic method is used for uploading blobs to the registry but some registries support a more efficient chunked method.
@@ -33,6 +36,10 @@ struct Push: AsyncParsableCommand {
   func run() async throws {
     let ociStorage = VMStorageOCI()
     let localVMDir = try VMStorageHelper.open(localName)
+    let lock = try localVMDir.lock()
+    if try !lock.trylock() {
+      throw RuntimeError.VMIsRunning(localName)
+    }
 
     // Parse remote names supplied by the user
     let remoteNames = try remoteNames.map{
@@ -73,7 +80,8 @@ struct Push: AsyncParsableCommand {
           registry: registry,
           references: references,
           chunkSizeMb: chunkSize,
-          diskFormat: diskFormat
+          diskFormat: diskFormat,
+          concurrency: concurrency
         )
         // Populate the local cache (if requested)
         if populateCache {

Sources/tart/Commands/Run.swift

@@ -12,6 +12,39 @@ var vm: VM?
 struct IPNotFound: Error {
 }
 
+@available(macOS 14, *)
+extension VZDiskSynchronizationMode {
+  public init(_ description: String) throws {
+    switch description {
+    case "none":
+      self = .none
+    case "full":
+      self = .full
+    case "":
+      self = .full
+    default:
+      throw RuntimeError.VMConfigurationError("unsupported disk synchronization mode: \"\(description)\"")
+    }
+  }
+}
+
+extension VZDiskImageSynchronizationMode {
+  public init(_ description: String) throws {
+    switch description {
+    case "none":
+      self = .none
+    case "fsync":
+      self = .fsync
+    case "full":
+      self = .full
+    case "":
+      self = .full
+    default:
+      throw RuntimeError.VMConfigurationError("unsupported disk image synchronization mode: \"\(description)\"")
+    }
+  }
+}
+
 struct Run: AsyncParsableCommand {
   static var configuration = CommandConfiguration(abstract: "Run a VM")
 
@@ -67,7 +100,7 @@ struct Run: AsyncParsableCommand {
   var vncExperimental: Bool = false
 
   @Option(help: ArgumentHelp("""
-  Additional disk attachments with an optional read-only specifier\n(e.g. --disk=\"disk.bin\" --disk=\"ubuntu.iso:ro\" --disk=\"/dev/disk0\" --disk "ghcr.io/cirruslabs/xcode:16.0:ro" --disk=\"nbd://localhost:10809/myDisk\")
+  Additional disk attachments with an optional read-only and synchronization options (e.g. --disk="disk.bin", --disk="ubuntu.iso:ro", --disk="/dev/disk0", --disk "ghcr.io/cirruslabs/xcode:16.0:ro" or --disk="nbd://localhost:10809/myDisk:sync=none")
   """, discussion: """
   The disk attachment can be a:
 
@@ -76,6 +109,12 @@ struct Run: AsyncParsableCommand {
   * remote VM name whose disk will be mounted
   * Network Block Device (NBD) URL
 
+  Options are comma-separated and are as follows:
+
+  * ro — attach the specified disk in read-only mode instead of the default read-write (e.g. --disk="disk.img:ro")
+
+  * sync=none — disable data synchronization with the permanent storage to increase performance at the cost of a higher chance of data loss (e.g. --disk="disk.img:sync=none")
+
   Learn how to create a disk image using Disk Utility here: https://support.apple.com/en-gb/guide/disk-utility/dskutl11888/mac
 
   To work with block devices, the easiest way is to modify their permissions (e.g. by using "sudo chown $USER /dev/diskX") or to run the Tart binary as root, which affects locating Tart VMs.
@@ -83,7 +122,7 @@ struct Run: AsyncParsableCommand {
   To work around this pass TART_HOME explicitly:
 
   sudo TART_HOME="$HOME/.tart" tart run sonoma --disk=/dev/disk0
-  """, valueName: "path[:ro]"))
+  """, valueName: "path[:options]"))
   var disk: [String] = []
 
   #if arch(arm64)
@@ -134,6 +173,20 @@ struct Run: AsyncParsableCommand {
   @Flag(help: ArgumentHelp("Restrict network access to the host-only network"))
   var netHost: Bool = false
 
+  @Option(help: ArgumentHelp("Set the root disk options (e.g. --root-disk-opts=\"ro\" or --root-disk-opts=\"sync=none\")",
+                             discussion: """
+                             Options are comma-separated and are as follows:
+
+                             * ro — attach the root disk in read-only mode instead of the default read-write (e.g. --root-disk-opts="ro")
+
+                             * sync=none — disable data synchronization with the permanent storage to increase performance at the cost of a higher chance of data loss (e.g. --root-disk-opts="sync=none")
+
+                             * sync=fsync — enable data synchronization with the permanent storage, but don't ensure that it was actually written (e.g. --root-disk-opts="sync=fsync")
+
+                             * sync=full — enable data synchronization with the permanent storage and ensure that it was actually written (e.g. --root-disk-opts="sync=full")
+                             """, valueName: "options"))
+  var rootDiskOpts: String = ""
+
   #if arch(arm64)
     @Flag(help: ArgumentHelp("Disables audio and entropy devices and switches to only Mac-specific input devices.", discussion: "Useful for running a VM that can be suspended via \"tart suspend\"."))
   #endif
@@ -213,8 +266,6 @@ struct Run: AsyncParsableCommand {
       try Softnet.configureSUIDBitIfNeeded()
     }
 
-    let additionalDiskAttachments = try additionalDiskAttachments()
-
     var serialPorts: [VZSerialPortConfiguration] = []
     if serial {
       let tty_fd = createPTY()
@@ -233,15 +284,19 @@ struct Run: AsyncParsableCommand {
       serialPorts.append(createSerialPortConfiguration(tty_read!, tty_write!))
     }
 
+    // Parse root disk options
+    let diskOptions = DiskOptions(rootDiskOpts)
+
     vm = try VM(
       vmDir: vmDir,
       network: userSpecifiedNetwork(vmDir: vmDir) ?? NetworkShared(),
-      additionalStorageDevices: additionalDiskAttachments,
+      additionalStorageDevices: try additionalDiskAttachments(),
       directorySharingDevices: directoryShares() + rosettaDirectoryShare(),
       serialPorts: serialPorts,
       suspendable: suspendable,
       audio: !noAudio,
-      clipboard: !noClipboard
+      clipboard: !noClipboard,
+      sync: VZDiskImageSynchronizationMode(diskOptions.syncModeRaw)
     )
 
     let vncImpl: VNC? = try {
@@ -449,90 +504,9 @@ struct Run: AsyncParsableCommand {
   }
 
   func additionalDiskAttachments() throws -> [VZStorageDeviceConfiguration] {
-    var result: [VZStorageDeviceConfiguration] = []
+    try disk.map {
-    let readOnlySuffix = ":ro"
+      try AdditionalDisk(parseFrom: $0).configuration
-    let expandedDiskPaths = disk.map { NSString(string:$0).expandingTildeInPath }
-
-    for rawDisk in expandedDiskPaths {
-      let diskReadOnly = rawDisk.hasSuffix(readOnlySuffix)
-      let diskPath = diskReadOnly ? String(rawDisk.prefix(rawDisk.count - readOnlySuffix.count)) : rawDisk
-
-      let diskURL = URL(string: diskPath)
-      if (["nbd", "nbds", "nbd+unix", "nbds+unix"].contains(diskURL?.scheme)) {
-        guard #available(macOS 14, *) else {
-          throw UnsupportedOSError("attaching Network Block Devices", "are")
-        }
-
-        let nbdAttachment = try VZNetworkBlockDeviceStorageDeviceAttachment(
-          url: diskURL!,
-          timeout: 30,
-          isForcedReadOnly: diskReadOnly,
-          synchronizationMode: VZDiskSynchronizationMode.none
-        )
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: nbdAttachment))
-        continue
-      }
-
-      let diskFileURL = URL(fileURLWithPath: diskPath)
-
-      if pathHasMode(diskPath, mode: S_IFBLK) {
-        guard #available(macOS 14, *) else {
-          throw UnsupportedOSError("attaching block devices", "are")
-        }
-
-        let fd = open(diskPath, diskReadOnly ? O_RDONLY : O_RDWR)
-        if fd == -1 {
-          let details = Errno(rawValue: CInt(errno))
-
-          switch details.rawValue {
-          case EBUSY:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "already in use, try umounting it via \"diskutil unmountDisk\" (when the whole disk) or \"diskutil umount\" (when mounting a single partition)")
-          case EACCES:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "permission denied, consider changing the disk's owner using \"sudo chown $USER \(diskFileURL.url.path)\" or run Tart as a superuser (see --disk help for more details on how to do that correctly)")
-          default:
-            throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "\(details)")
-          }
-        }
-
-        let blockAttachment = try VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: FileHandle(fileDescriptor: fd, closeOnDealloc: true),
-                                                                           readOnly: diskReadOnly, synchronizationMode: .full)
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: blockAttachment))
-        continue
-      }
-
-      // Support remote VM names in --disk command-line argument
-      if let remoteName = try? RemoteName(diskPath) {
-        let vmDir = try VMStorageOCI().open(remoteName)
-
-        // Unfortunately, VZDiskImageStorageDeviceAttachment does not support
-        // FileHandle, so we can't easily clone the disk, open it and unlink(2)
-        // to simplify the garbage collection, so use an intermediate directory.
-        let clonedDiskURL = try Config().tartTmpDir.appendingPathComponent("run-disk-\(UUID().uuidString)")
-
-        try FileManager.default.copyItem(at: vmDir.diskURL, to: clonedDiskURL)
-
-        let lock = try FileLock(lockURL: clonedDiskURL)
-        try lock.lock()
-
-        let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: clonedDiskURL, readOnly: diskReadOnly)
-        result.append(VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment))
-        continue
-      }
-
-      // Error out if the disk is locked by the host (e.g. it was mounted in Finder),
-      // see https://github.com/cirruslabs/tart/issues/323 for more details.
-      if try !diskReadOnly && !FileLock(lockURL: diskFileURL).trylock() {
-        throw RuntimeError.DiskAlreadyInUse("disk \(diskFileURL.url.path) seems to be already in use, unmount it first in Finder")
-      }
-
-      let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(
-        url: diskFileURL,
-        readOnly: diskReadOnly
-      )
-      result.append(VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment))
     }
-
-    return result
   }
 
   func directoryShares() throws -> [VZDirectorySharingDeviceConfiguration] {
@@ -752,6 +726,134 @@ struct VMView: NSViewRepresentable {
   }
 }
 
+struct AdditionalDisk {
+  let configuration: VZStorageDeviceConfiguration
+
+  init(parseFrom: String) throws {
+    let (diskPath, readOnly, syncModeRaw) = Self.parseOptions(parseFrom)
+
+    self.configuration = try Self.craft(diskPath, readOnly: readOnly, syncModeRaw: syncModeRaw)
+  }
+
+  static func craft(_ diskPath: String, readOnly diskReadOnly: Bool, syncModeRaw: String) throws -> VZStorageDeviceConfiguration {
+    let diskURL = URL(string: diskPath)
+
+    if (["nbd", "nbds", "nbd+unix", "nbds+unix"].contains(diskURL?.scheme)) {
+      guard #available(macOS 14, *) else {
+        throw UnsupportedOSError("attaching Network Block Devices", "are")
+      }
+
+      let nbdAttachment = try VZNetworkBlockDeviceStorageDeviceAttachment(
+        url: diskURL!,
+        timeout: 30,
+        isForcedReadOnly: diskReadOnly,
+        synchronizationMode: try VZDiskSynchronizationMode(syncModeRaw)
+      )
+
+      return VZVirtioBlockDeviceConfiguration(attachment: nbdAttachment)
+    }
+
+    // Expand the tilde (~) since at this point we're dealing with a local path,
+    // and "expandingTildeInPath" seems to corrupt the remote URLs like nbd://
+    let diskPath = NSString(string: diskPath).expandingTildeInPath
+
+    let diskFileURL = URL(fileURLWithPath: diskPath)
+
+    if pathHasMode(diskPath, mode: S_IFBLK) {
+      guard #available(macOS 14, *) else {
+        throw UnsupportedOSError("attaching block devices", "are")
+      }
+
+      let fd = open(diskPath, diskReadOnly ? O_RDONLY : O_RDWR)
+      if fd == -1 {
+        let details = Errno(rawValue: CInt(errno))
+
+        switch details.rawValue {
+        case EBUSY:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "already in use, try umounting it via \"diskutil unmountDisk\" (when the whole disk) or \"diskutil umount\" (when mounting a single partition)")
+        case EACCES:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "permission denied, consider changing the disk's owner using \"sudo chown $USER \(diskFileURL.url.path)\" or run Tart as a superuser (see --disk help for more details on how to do that correctly)")
+        default:
+          throw RuntimeError.FailedToOpenBlockDevice(diskFileURL.url.path, "\(details)")
+        }
+      }
+
+      let blockAttachment = try VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: FileHandle(fileDescriptor: fd, closeOnDealloc: true),
+                                                                         readOnly: diskReadOnly, synchronizationMode: try VZDiskSynchronizationMode(syncModeRaw))
+
+      return VZVirtioBlockDeviceConfiguration(attachment: blockAttachment)
+    }
+
+    // Support remote VM names in --disk command-line argument
+    if let remoteName = try? RemoteName(diskPath) {
+      let vmDir = try VMStorageOCI().open(remoteName)
+
+      // Unfortunately, VZDiskImageStorageDeviceAttachment does not support
+      // FileHandle, so we can't easily clone the disk, open it and unlink(2)
+      // to simplify the garbage collection, so use an intermediate directory.
+      let clonedDiskURL = try Config().tartTmpDir.appendingPathComponent("run-disk-\(UUID().uuidString)")
+
+      try FileManager.default.copyItem(at: vmDir.diskURL, to: clonedDiskURL)
+
+      let lock = try FileLock(lockURL: clonedDiskURL)
+      try lock.lock()
+
+      let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: clonedDiskURL, readOnly: diskReadOnly)
+
+      return VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment)
+    }
+
+    // Error out if the disk is locked by the host (e.g. it was mounted in Finder),
+    // see https://github.com/cirruslabs/tart/issues/323 for more details.
+    if try !diskReadOnly && !FileLock(lockURL: diskFileURL).trylock() {
+      throw RuntimeError.DiskAlreadyInUse("disk \(diskFileURL.url.path) seems to be already in use, unmount it first in Finder")
+    }
+
+    let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(
+      url: diskFileURL,
+      readOnly: diskReadOnly,
+      cachingMode: .automatic,
+      synchronizationMode: try VZDiskImageSynchronizationMode(syncModeRaw)
+    )
+
+    return VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment)
+  }
+
+  static func parseOptions(_ parseFrom: String) -> (String, Bool, String) {
+    var arguments = parseFrom.split(separator: ":")
+
+    let options = DiskOptions(String(arguments.last!))
+    if options.foundAtLeastOneOption {
+      arguments.removeLast()
+    }
+
+    return (arguments.joined(separator: ":"), options.readOnly, options.syncModeRaw)
+  }
+}
+
+struct DiskOptions {
+  var readOnly: Bool = false
+  var syncModeRaw: String = ""
+  var foundAtLeastOneOption: Bool = false
+
+  init(_ parseFrom: String) {
+    let options = parseFrom.split(separator: ",")
+
+    for option in options {
+      switch true {
+      case option == "ro":
+        self.readOnly = true
+        self.foundAtLeastOneOption = true
+      case option.hasPrefix("sync="):
+        self.syncModeRaw = String(option.dropFirst("sync=".count))
+        self.foundAtLeastOneOption = true
+      default:
+        continue
+      }
+    }
+  }
+}
+
 struct DirectoryShare {
   let name: String?
   let path: URL

Sources/tart/LocalLayerCache.swift

@@ -1,10 +1,25 @@
 import Foundation
 
 struct LocalLayerCache {
+  struct DigestInfo {
+    let range: Range<Data.Index>
+    let compressedDigest: String
+    let uncompressedContentDigest: String?
+  }
+
+  let name: String
+  let deduplicatedBytes: UInt64
+  let diskURL: URL
+
   private let mappedDisk: Data
-  private var digestToRange: [String : Range<Data.Index>] = [:]
+  private var digestToRange: [String: DigestInfo] = [:]
+  private var offsetToRange: [UInt64: DigestInfo] = [:]
+
+  init?(_ name: String, _ deduplicatedBytes: UInt64, _ diskURL: URL, _ manifest: OCIManifest) throws {
+    self.name = name
+    self.deduplicatedBytes = deduplicatedBytes
+    self.diskURL = diskURL
 
-  init?(_ diskURL: URL, _ manifest: OCIManifest) throws {
     // mmap(2) the disk that contains the layers from the manifest
     self.mappedDisk = try Data(contentsOf: diskURL, options: [.alwaysMapped])
 
@@ -16,17 +31,27 @@ struct LocalLayerCache {
         return nil
       }
 
-      self.digestToRange[layer.digest] = Int(offset)..<Int(offset+uncompressedSize)
+      let info = DigestInfo(
+        range: Int(offset)..<Int(offset + uncompressedSize),
+        compressedDigest: layer.digest,
+        uncompressedContentDigest: layer.uncompressedContentDigest()!
+      )
+      self.digestToRange[layer.digest] = info
+      self.offsetToRange[offset] = info
 
       offset += uncompressedSize
     }
   }
 
-  func find(_ digest: String) -> Data? {
+  func findInfo(digest: String, offsetHint: UInt64) -> DigestInfo? {
-    guard let foundRange = self.digestToRange[digest] else {
+    // Layers can have the same digests, for example, empty ones. Let's use the offset hint to make a better guess.
-      return nil
+    if let info = self.offsetToRange[offsetHint], info.compressedDigest == digest {
+      return info
     }
+    return self.digestToRange[digest]
+  }
 
-    return self.mappedDisk.subdata(in: foundRange)
+  func subdata(_ range: Range<Data.Index>) -> Data {
+    return self.mappedDisk.subdata(in: range)
   }
 }

Sources/tart/OCI/Layerizer/Disk.swift

@@ -1,6 +1,6 @@
 import Foundation
 
 protocol Disk {
-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer]
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer]
   static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache?) async throws
 }

Sources/tart/OCI/Layerizer/DiskV1.swift

@@ -5,7 +5,7 @@ class DiskV1: Disk {
   private static let bufferSizeBytes = 4 * 1024 * 1024
   private static let layerLimitBytes = 500 * 1000 * 1000
 
-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer] {
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer] {
     var pushedLayers: [OCIManifestLayer] = []
 
     // Open the disk file

Sources/tart/OCI/Layerizer/DiskV2.swift

@@ -1,44 +1,74 @@
 import Foundation
 import Compression
+import System
 
 class DiskV2: Disk {
   private static let bufferSizeBytes = 4 * 1024 * 1024
-  private static let layerLimitBytes = 500 * 1000 * 1000
+  private static let layerLimitBytes = 512 * 1024 * 1024
 
-  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, progress: Progress) async throws -> [OCIManifestLayer] {
+  static func push(diskURL: URL, registry: Registry, chunkSizeMb: Int, concurrency: UInt, progress: Progress) async throws -> [OCIManifestLayer] {
-    var pushedLayers: [OCIManifestLayer] = []
+    var pushedLayers: [(index: Int, pushedLayer: OCIManifestLayer)] = []
 
     // Open the disk file
     let mappedDisk = try Data(contentsOf: diskURL, options: [.alwaysMapped])
 
     // Compress the disk file as multiple individually decompressible streams,
     // each equal ``Self.layerLimitBytes`` bytes or less due to LZ4 compression
-    for data in mappedDisk.chunks(ofCount: layerLimitBytes) {
+    try await withThrowingTaskGroup(of: (Int, OCIManifestLayer).self) { group in
-      let compressedData = try (data as NSData).compressed(using: .lz4) as Data
+      for (index, data) in mappedDisk.chunks(ofCount: layerLimitBytes).enumerated() {
+        // Respect the concurrency limit
+        if index >= concurrency {
+          if let (index, pushedLayer) = try await group.next() {
+            pushedLayers.append((index, pushedLayer))
+          }
+        }
 
-      let layerDigest = try await registry.pushBlob(fromData: compressedData, chunkSizeMb: chunkSizeMb)
+        // Launch a disk layer pushing task
+        group.addTask {
+          let compressedData = try (data as NSData).compressed(using: .lz4) as Data
 
-      pushedLayers.append(OCIManifestLayer(
+          let layerDigest = try await registry.pushBlob(fromData: compressedData, chunkSizeMb: chunkSizeMb)
-        mediaType: diskV2MediaType,
-        size: compressedData.count,
-        digest: layerDigest,
-        uncompressedSize: UInt64(data.count),
-        uncompressedContentDigest: Digest.hash(data)
-      ))
 
-      // Update progress using a relative value
+          // Update progress using a relative value
-      progress.completedUnitCount += Int64(data.count)
+          progress.completedUnitCount += Int64(data.count)
+
+          return (index, OCIManifestLayer(
+            mediaType: diskV2MediaType,
+            size: compressedData.count,
+            digest: layerDigest,
+            uncompressedSize: UInt64(data.count),
+            uncompressedContentDigest: Digest.hash(data)
+          ))
+        }
+      }
+
+      for try await pushedLayer in group {
+        pushedLayers.append(pushedLayer)
+      }
     }
 
-    return pushedLayers
+    return pushedLayers.sorted {
+      $0.index < $1.index
+    }.map {
+      $0.pushedLayer
+    }
   }
 
   static func pull(registry: Registry, diskLayers: [OCIManifestLayer], diskURL: URL, concurrency: UInt, progress: Progress, localLayerCache: LocalLayerCache? = nil) async throws {
     // Support resumable pulls
     let pullResumed = FileManager.default.fileExists(atPath: diskURL.path)
 
-    if !pullResumed && !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
+    if !pullResumed {
-      throw OCIError.FailedToCreateVmFile
+      if let localLayerCache = localLayerCache {
+        // Clone the local layer cache's disk and use it as a base, potentially
+        // reducing the space usage since some blocks won't be written at all
+        try FileManager.default.copyItem(at: localLayerCache.diskURL, to: diskURL)
+      } else {
+        // Otherwise create an empty disk
+        if !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
+          throw OCIError.FailedToCreateVmFile
+        }
+      }
     }
 
     // Calculate the uncompressed disk size
@@ -58,6 +88,15 @@ class DiskV2: Disk {
     try disk.truncate(atOffset: uncompressedDiskSize)
     try disk.close()
 
+    // Determine the file system block size
+    var st = stat()
+    if stat(diskURL.path, &st) == -1 {
+      let details = Errno(rawValue: errno)
+
+      throw RuntimeError.PullFailed("failed to stat(2) disk \(diskURL.path): \(details)")
+    }
+    let fsBlockSize = UInt64(st.st_blksize)
+
     // Concurrently fetch and decompress layers
     try await withThrowingTaskGroup(of: Void.self) { group in
       var globalDiskWritingOffset: UInt64 = 0
@@ -82,20 +121,39 @@ class DiskV2: Disk {
         // Launch a fetching and decompression task
         group.addTask {
           // No need to fetch and decompress anything if we've already done so
-          if try pullResumed && Digest.hash(diskURL, offset: diskWritingOffset, size: uncompressedLayerSize) == uncompressedLayerContentDigest {
+          if pullResumed {
-            // Update the progress
+            // do not check hash in the condition above to make it lazy e.g. only do expensive calculations if needed
-            progress.completedUnitCount += Int64(diskLayer.size)
+            if try Digest.hash(diskURL, offset: diskWritingOffset, size: uncompressedLayerSize) == uncompressedLayerContentDigest {
+              // Update the progress
+              progress.completedUnitCount += Int64(diskLayer.size)
 
-            return
+              return
+            }
           }
 
-          // Open the disk file
+          // Open the disk file for writing
           let disk = try FileHandle(forWritingTo: diskURL)
 
+          // Also open the disk file for reading and verifying
+          // its contents in case the local layer cache is used
+          let rdisk: FileHandle? = if localLayerCache != nil {
+            try FileHandle(forReadingFrom: diskURL)
+          } else {
+            nil
+          }
+
           // Check if we already have this layer contents in the local layer cache
-          if let localLayerCache = localLayerCache, let data = localLayerCache.find(diskLayer.digest), Digest.hash(data) == uncompressedLayerContentDigest {
+          if let localLayerCache = localLayerCache, let localLayerInfo = localLayerCache.findInfo(digest: diskLayer.digest, offsetHint: diskWritingOffset) {
-            // Fulfil the layer contents from the local blob cache
+            // indicates that the locally cloned disk image has the same content at the given offset
-            _ = try zeroSkippingWrite(disk, diskWritingOffset, data)
+            let localHit = localLayerInfo.uncompressedContentDigest == uncompressedLayerContentDigest
+              && localLayerInfo.range.lowerBound == diskWritingOffset
+            // doesn't seem that localHit can ever be false if the localLayerCache is not nil
+            // but let's just add extra safety here and check it
+            if !localHit {
+              // Fulfil the layer contents from the local blob cache
+              let data = localLayerCache.subdata(localLayerInfo.range)
+              _ = try zeroSkippingWrite(disk, rdisk, fsBlockSize, diskWritingOffset, data)
+            }
             try disk.close()
 
             // Update the progress
@@ -112,7 +170,7 @@ class DiskV2: Disk {
               return
             }
 
-            diskWritingOffset = try zeroSkippingWrite(disk, diskWritingOffset, data)
+            diskWritingOffset = try zeroSkippingWrite(disk, rdisk, fsBlockSize, diskWritingOffset, data)
           }
 
           try await registry.pullBlob(diskLayer.digest) { data in
@@ -132,7 +190,7 @@ class DiskV2: Disk {
     }
   }
 
-  private static func zeroSkippingWrite(_ disk: FileHandle, _ offset: UInt64, _ data: Data) throws -> UInt64 {
+  private static func zeroSkippingWrite(_ disk: FileHandle, _ rdisk: FileHandle?, _ fsBlockSize: UInt64, _ offset: UInt64, _ data: Data) throws -> UInt64 {
     let holeGranularityBytes = 64 * 1024
 
     // A zero chunk for faster than byte-by-byte comparisons
@@ -152,7 +210,38 @@ class DiskV2: Disk {
     var offset = offset
 
     for chunk in data.chunks(ofCount: holeGranularityBytes) {
-      // Only write chunks that are not zero
+      // If the local layer cache is used, only write chunks that differ
+      // since the base disk can contain anything at any position
+      if let rdisk = rdisk {
+        // F_PUNCHHOLE requires the holes to be aligned to file system block boundaries
+        let isHoleAligned = (offset % fsBlockSize) == 0 && (UInt64(chunk.count) % fsBlockSize) == 0
+
+        if isHoleAligned && chunk == zeroChunk {
+          var arg = fpunchhole_t(fp_flags: 0, reserved: 0, fp_offset: off_t(offset), fp_length: off_t(chunk.count))
+
+          if fcntl(disk.fileDescriptor, F_PUNCHHOLE, &arg) == -1 {
+            let details = Errno(rawValue: errno)
+
+            throw RuntimeError.PullFailed("failed to punch hole: \(details)")
+          }
+        } else {
+          try rdisk.seek(toOffset: offset)
+          let actualContentsOnDisk = try rdisk.read(upToCount: chunk.count)
+
+          if chunk != actualContentsOnDisk {
+            try disk.seek(toOffset: offset)
+            disk.write(chunk)
+          }
+        }
+
+        offset += UInt64(chunk.count)
+
+        continue
+      }
+
+      // Otherwise, only write chunks that are not zero
+      // since the base disk is created from scratch and
+      // is zeroed via truncate(2)
       if chunk != zeroChunk {
         try disk.seek(toOffset: offset)
         disk.write(chunk)

Sources/tart/OCI/Manifest.swift

@@ -78,7 +78,7 @@ struct OCIManifestConfig: Codable, Equatable {
   var digest: String
 }
 
-struct OCIManifestLayer: Codable, Equatable {
+struct OCIManifestLayer: Codable, Equatable, Hashable {
   var mediaType: String
   var size: Int
   var digest: String
@@ -113,6 +113,14 @@ struct OCIManifestLayer: Codable, Equatable {
   func uncompressedContentDigest() -> String? {
     annotations?[uncompressedContentDigestAnnotation]
   }
+
+  static func == (lhs: Self, rhs: Self) -> Bool {
+    return lhs.digest == rhs.digest
+  }
+
+  func hash(into hasher: inout Hasher) {
+    hasher.combine(digest)
+  }
 }
 
 struct Descriptor: Equatable {

Sources/tart/OCI/Registry.swift

@@ -112,11 +112,11 @@ class Registry {
     return host
   }
 
-  init(urlComponents: URLComponents,
+  init(baseURL: URL,
        namespace: String,
        credentialsProviders: [CredentialsProvider] = [EnvironmentCredentialsProvider(), DockerConfigCredentialsProvider(), KeychainCredentialsProvider()]
   ) throws {
-    baseURL = urlComponents.url!
+    self.baseURL = baseURL
     self.namespace = namespace
     self.credentialsProviders = credentialsProviders
   }
@@ -130,7 +130,17 @@ class Registry {
     let proto = insecure ? "http" : "https"
     let baseURLComponents = URLComponents(string: proto + "://" + host + "/v2/")!
 
-    try self.init(urlComponents: baseURLComponents, namespace: namespace, credentialsProviders: credentialsProviders)
+    guard let baseURL = baseURLComponents.url else {
+      var hint = ""
+
+      if host.hasPrefix("http://") || host.hasPrefix("https://") {
+        hint += ", make sure that it doesn't start with http:// or https://"
+      }
+
+      throw RuntimeError.ImproperlyFormattedHost(host, hint)
+    }
+
+    try self.init(baseURL: baseURL, namespace: namespace, credentialsProviders: credentialsProviders)
   }
 
   func ping() async throws {

Sources/tart/URL+Prunable.swift

@@ -1,4 +1,5 @@
 import Foundation
+import XAttr
 
 extension URL: Prunable {
   var url: URL {
@@ -13,7 +14,31 @@ extension URL: Prunable {
     try resourceValues(forKeys: [.totalFileAllocatedSizeKey]).totalFileAllocatedSize!
   }
 
+  func deduplicatedSizeBytes() throws -> Int {
+    let values = try resourceValues(forKeys: [.totalFileAllocatedSizeKey, .mayShareFileContentKey])
+    // make sure the file's origin file is there and duplication works
+    if values.mayShareFileContent == true {
+      return Int(deduplicatedBytes())
+    }
+    return 0
+  }
+
   func sizeBytes() throws -> Int {
     try resourceValues(forKeys: [.totalFileSizeKey]).totalFileSize!
   }
+
+  func setDeduplicatedBytes(_ size: UInt64) {
+    let data = "\(size)".data(using: .utf8)!
+    try! self.setExtendedAttribute(name: "run.tart.deduplicated-bytes", value: data)
+  }
+
+  func deduplicatedBytes() -> UInt64 {
+    guard let data = try? self.extendedAttributeValue(forName: "run.tart.deduplicated-bytes") else {
+      return 0
+    }
+    if let strValue = String(data: data, encoding: .utf8) {
+      return UInt64(strValue) ?? 0
+    }
+    return 0
+  }
 }

Sources/tart/VM.swift

@@ -48,7 +48,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
        serialPorts: [VZSerialPortConfiguration] = [],
        suspendable: Bool = false,
        audio: Bool = true,
-       clipboard: Bool = true
+       clipboard: Bool = true,
+       sync: VZDiskImageSynchronizationMode = .full
   ) throws {
     name = vmDir.name
     config = try VMConfig.init(fromURL: vmDir.configURL)
@@ -66,7 +67,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
                                                 serialPorts: serialPorts,
                                                 suspendable: suspendable,
                                                 audio: audio,
-                                                clipboard: clipboard
+                                                clipboard: clipboard,
+                                                sync: sync
     )
     virtualMachine = VZVirtualMachine(configuration: configuration)
 
@@ -294,7 +296,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     serialPorts: [VZSerialPortConfiguration],
     suspendable: Bool = false,
     audio: Bool = true,
-    clipboard: Bool = true
+    clipboard: Bool = true,
+    sync: VZDiskImageSynchronizationMode = .full
   ) throws -> VZVirtualMachineConfiguration {
     let configuration = VZVirtualMachineConfiguration()
 
@@ -312,16 +315,19 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     configuration.graphicsDevices = [vmConfig.platform.graphicsDevice(vmConfig: vmConfig)]
 
     // Audio
+    let soundDeviceConfiguration = VZVirtioSoundDeviceConfiguration()
+
+    let inputAudioStreamConfiguration = VZVirtioSoundDeviceInputStreamConfiguration()
+    let outputAudioStreamConfiguration = VZVirtioSoundDeviceOutputStreamConfiguration()
+
     if audio && !suspendable {
-      let soundDeviceConfiguration = VZVirtioSoundDeviceConfiguration()
-      let inputAudioStreamConfiguration = VZVirtioSoundDeviceInputStreamConfiguration()
       inputAudioStreamConfiguration.source = VZHostAudioInputStreamSource()
-      let outputAudioStreamConfiguration = VZVirtioSoundDeviceOutputStreamConfiguration()
       outputAudioStreamConfiguration.sink = VZHostAudioOutputStreamSink()
-      soundDeviceConfiguration.streams = [inputAudioStreamConfiguration, outputAudioStreamConfiguration]
-      configuration.audioDevices = [soundDeviceConfiguration]
     }
 
+    soundDeviceConfiguration.streams = [inputAudioStreamConfiguration, outputAudioStreamConfiguration]
+    configuration.audioDevices = [soundDeviceConfiguration]
+
     // Keyboard and mouse
     configuration.keyboards = vmConfig.platform.keyboards()
     configuration.pointingDevices = vmConfig.platform.pointingDevices()
@@ -347,8 +353,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
     // Storage
     let attachment: VZDiskImageStorageDeviceAttachment = vmConfig.os == .linux ?
       // Use "cached" caching mode for virtio drive to prevent fs corruption on linux
-      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .cached, synchronizationMode: .full) :
+      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .cached, synchronizationMode: sync) :
-      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false)
+      try VZDiskImageStorageDeviceAttachment(url: diskURL, readOnly: false, cachingMode: .automatic, synchronizationMode: sync)
 
     var device: VZStorageDeviceConfiguration
     if #available(macOS 14, *), vmConfig.os == .linux {

Sources/tart/VMDirectory+OCI.swift

@@ -11,9 +11,6 @@ enum OCIError: Error {
 }
 
 extension VMDirectory {
-  private static let bufferSizeBytes = 64 * 1024 * 1024
-  private static let layerLimitBytes = 500 * 1000 * 1000
-
   func pullFromRegistry(registry: Registry, manifest: OCIManifest, concurrency: UInt, localLayerCache: LocalLayerCache?) async throws {
     // Pull VM's config file layer and re-serialize it into a config file
     let configLayers = manifest.layers.filter {
@@ -62,6 +59,11 @@ extension VMDirectory {
       throw RuntimeError.PullFailed("failed to decompress disk: \(error.localizedDescription)")
     }
 
+    if let llc = localLayerCache {
+      // set custom attribute to remember deduplicated bytes
+      diskURL.setDeduplicatedBytes(llc.deduplicatedBytes)
+    }
+
     // Pull VM's NVRAM file layer and store it in an NVRAM file
     defaultLogger.appendNewLine("pulling NVRAM...")
 
@@ -79,9 +81,12 @@ extension VMDirectory {
       nvram.write(data)
     }
     try nvram.close()
+
+    // Serialize VM's manifest to enable better deduplication on subsequent "tart pull"'s
+    try manifest.toJSON().write(to: manifestURL)
   }
 
-  func pushToRegistry(registry: Registry, references: [String], chunkSizeMb: Int, diskFormat: String) async throws -> RemoteName {
+  func pushToRegistry(registry: Registry, references: [String], chunkSizeMb: Int, diskFormat: String, concurrency: UInt) async throws -> RemoteName {
     var layers = Array<OCIManifestLayer>()
 
     // Read VM's config and push it as blob
@@ -100,9 +105,9 @@ extension VMDirectory {
 
     switch diskFormat {
     case "v1":
-      layers.append(contentsOf: try await DiskV1.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, progress: progress))
+      layers.append(contentsOf: try await DiskV1.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, concurrency: concurrency, progress: progress))
     case "v2":
-      layers.append(contentsOf: try await DiskV2.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, progress: progress))
+      layers.append(contentsOf: try await DiskV2.push(diskURL: diskURL, registry: registry, chunkSizeMb: chunkSizeMb, concurrency: concurrency, progress: progress))
     default:
       throw RuntimeError.OCIUnsupportedDiskFormat(diskFormat)
     }

Sources/tart/VMDirectory.swift

@@ -23,6 +23,9 @@ struct VMDirectory: Prunable {
   var stateURL: URL {
     baseURL.appendingPathComponent("state.vzvmsave")
   }
+  var manifestURL: URL {
+    baseURL.appendingPathComponent("manifest.json")
+  }
 
   var explicitlyPulledMark: URL {
     baseURL.appendingPathComponent(".explicitly-pulled")
@@ -179,6 +182,14 @@ struct VMDirectory: Prunable {
     try allocatedSizeBytes() / 1000 / 1000 / 1000
   }
 
+  func deduplicatedSizeBytes() throws -> Int {
+    try configURL.deduplicatedSizeBytes() + diskURL.deduplicatedSizeBytes() + nvramURL.deduplicatedSizeBytes()
+  }
+
+  func deduplicatedSizeGB() throws -> Int {
+    try deduplicatedSizeBytes() / 1000 / 1000 / 1000
+  }
+
   func sizeBytes() throws -> Int {
     try configURL.sizeBytes() + diskURL.sizeBytes() + nvramURL.sizeBytes()
   }

Sources/tart/VMStorageHelper.swift

@@ -61,6 +61,7 @@ enum RuntimeError : Error {
   case PIDLockFailed(_ message: String)
   case FailedToParseRemoteName(_ message: String)
   case VMTerminationFailed(_ message: String)
+  case ImproperlyFormattedHost(_ host: String, _ hint: String)
   case InvalidCredentials(_ message: String)
   case VMDirectoryAlreadyInitialized(_ message: String)
   case ExportFailed(_ message: String)
@@ -107,6 +108,8 @@ extension RuntimeError : CustomStringConvertible {
       return "failed to parse remote name: \(cause)"
     case .VMTerminationFailed(let message):
       return message
+    case .ImproperlyFormattedHost(let host, let hint):
+      return "improperly formatted host \"\(host)\" was provided\(hint)"
     case .InvalidCredentials(let message):
       return message
     case .VMDirectoryAlreadyInitialized(let message):

Sources/tart/VMStorageLocal.swift

@@ -66,7 +66,7 @@ class VMStorageLocal: PrunableStorage {
   }
 
   func prunables() throws -> [Prunable] {
-    try list().map { (_, vmDir) in vmDir }
+    try list().map { (_, vmDir) in vmDir }.filter { try !$0.running() }
   }
 
   func hasVMsWithMACAddress(macAddress: String) throws -> Bool {

Sources/tart/VMStorageOCI.swift

@@ -197,17 +197,21 @@ class VMStorageOCI: PrunableStorage {
 
       try await withTaskCancellationHandler(operation: {
         try await retry(maxAttempts: 5, backoff: .exponentialWithFullJitter(baseDelay: .seconds(5), maxDelay: .seconds(60))) {
-          var localLayerCache: LocalLayerCache? = nil
+          // Choose the best base image which has the most deduplication ratio
+          let localLayerCache = try await chooseLocalLayerCache(name, manifest, registry)
 
-          if name.reference.type == .Tag,
+          if let llc = localLayerCache {
-             let vmDir = try? open(name),
+            let deduplicatedHuman = ByteCountFormatter.string(fromByteCount: Int64(llc.deduplicatedBytes), countStyle: .file)
-             let digest = try? digest(name),
+
-             let (manifest, _) = try? await registry.pullManifest(reference: digest) {
+            defaultLogger.appendNewLine("found an image \(llc.name) that will allow us to deduplicate \(deduplicatedHuman), using it as a base...")
-            localLayerCache = try LocalLayerCache(vmDir.diskURL, manifest)
           }
 
           try await tmpVMDir.pullFromRegistry(registry: registry, manifest: manifest, concurrency: concurrency, localLayerCache: localLayerCache)
         } recoverFromFailure: { error in
+          if error is RuntimeError {
+            return .throw
+          }
+
           print("Error: \(error.localizedDescription)")
           print("Attempting to re-try...")
 
@@ -249,6 +253,59 @@ class VMStorageOCI: PrunableStorage {
 
     try gc()
   }
+
+  func chooseLocalLayerCache(_ name: RemoteName, _ manifest: OCIManifest, _ registry: Registry) async throws -> LocalLayerCache? {
+    // Establish a closure that will calculate how much bytes
+    // we'll deduplicate if we re-use the given manifest
+    let target = Swift.Set(manifest.layers)
+
+    let calculateDeduplicatedBytes = { (manifest: OCIManifest) -> UInt64 in
+      target.intersection(manifest.layers).map({ UInt64($0.size) }).reduce(0, +)
+    }
+
+    // Load OCI VM images and their manifests (if present)
+    var candidates: [(name: String, vmDir: VMDirectory, manifest: OCIManifest, deduplicatedBytes: UInt64)] = []
+
+    for (name, vmDir, isSymlink) in try list() {
+      if isSymlink {
+        continue
+      }
+
+      guard let manifestJSON = try? Data(contentsOf: vmDir.manifestURL) else {
+        continue
+      }
+
+      guard let manifest = try? OCIManifest(fromJSON: manifestJSON) else {
+        continue
+      }
+
+      candidates.append((name, vmDir, manifest, calculateDeduplicatedBytes(manifest)))
+    }
+
+    // Previously we haven't stored the OCI VM image manifests, but still fetched the VM image manifest if
+    // what the user was trying to pull was a tagged image, and we already had that image in the OCI VM cache
+    //
+    // Keep supporting this behavior for backwards comaptibility, but only communicate
+    // with the registry if we haven't already retrieved the manifest for that OCI VM image.
+    if name.reference.type == .Tag,
+       let vmDir = try? open(name),
+       let digest = try? digest(name),
+       try !candidates.contains(where: {try $0.manifest.digest() == digest}),
+       let (manifest, _) = try? await registry.pullManifest(reference: digest) {
+      candidates.append((name.description, vmDir, manifest, calculateDeduplicatedBytes(manifest)))
+    }
+
+    // Now, find the best match based on how many bytes we'll deduplicate
+    let choosen = candidates.filter {
+      $0.deduplicatedBytes > 1024 * 1024 * 1024 // save at least 1GB
+    }.max { left, right in
+      return left.deduplicatedBytes < right.deduplicatedBytes
+    }
+
+    return try choosen.flatMap({ choosen in
+      try LocalLayerCache(choosen.name, choosen.deduplicatedBytes, choosen.vmDir.diskURL, choosen.manifest)
+    })
+  }
 }
 
 extension URL {

Tests/TartTests/LayerizerTests.swift

@@ -36,7 +36,7 @@ final class LayerizerTests: XCTestCase {
     let pulledDiskFileURL = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
 
     print("pushing disk...")
-    let diskLayers = try await DiskV1.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, progress: Progress())
+    let diskLayers = try await DiskV1.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, concurrency: 4, progress: Progress())
 
     print("pulling disk...")
     try await DiskV1.pull(registry: registry, diskLayers: diskLayers, diskURL: pulledDiskFileURL, concurrency: 16, progress: Progress())
@@ -57,7 +57,7 @@ final class LayerizerTests: XCTestCase {
     let pulledDiskFileURL = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
 
     print("pushing disk...")
-    let diskLayers = try await DiskV2.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, progress: Progress())
+    let diskLayers = try await DiskV2.push(diskURL: originalDiskFileURL, registry: registry, chunkSizeMb: 0, concurrency: 4, progress: Progress())
 
     print("pulling disk...")
     try await DiskV2.pull(registry: registry, diskLayers: diskLayers, diskURL: pulledDiskFileURL, concurrency: 16, progress: Progress())

Tests/TartTests/Util/RegistryRunner.swift

@@ -39,7 +39,7 @@ class RegistryRunner {
     let port = try Self.dockerCmd("inspect", containerID, "--format", "{{(index (index .NetworkSettings.Ports \"5000/tcp\") 0).HostPort}}")
       .trimmingCharacters(in: CharacterSet.newlines)
 
-    registry = try Registry(urlComponents: URLComponents(string: "http://127.0.0.1:\(port)/v2/")!,
+    registry = try Registry(baseURL: URL(string: "http://127.0.0.1:\(port)/v2/")!,
                             namespace: "vm-image")
 
     // Wait for the Docker Registry to start

docs/theme/overrides/home.html

@@ -85,10 +85,10 @@
       <!-- landing image -->
       <div class="tx-landing__image">
         <dotlottie-player
-          src="/assets/animations/TartLogo.lottie"
+            src="/assets/animations/TartLogo.lottie"
-          mode="normal"
+            mode="normal"
-          style="width: 75%; margin: auto"
+            style="width: 75%; margin: auto"
-          autoplay
+            autoplay
         />
       </div>
 
@@ -114,10 +114,9 @@
       <header class="md-typeset">
         <h1 id="virtualization-and-beyond">
           Virtualization and beyond
-          <a
+          <a href="#virtualization-and-beyond"
-            href="#virtualization-and-beyond"
+             class="headerlink"
-            class="headerlink"
+             title="Permanent link"
-            title="Permanent link"
           >
             ¶
           </a>
@@ -125,12 +124,11 @@
       </header>
       <div class="mdx-spotlight">
         <figure class="mdx-spotlight__feature">
-          <img
+          <img src="assets/images/spotlight/virtualization-framework.webp"
-            src="assets/images/spotlight/virtualization-framework.webp"
+               alt="Apple’s native Virtualization.Framework"
-            alt="Apple’s native Virtualization.Framework"
+               loading="lazy"
-            loading="lazy"
+               width="500"
-            width="500"
+               height="212"
-            height="212"
           />
           <figcaption class="md-typeset">
             <h2>Native performance</h2>
@@ -144,12 +142,11 @@
           </figcaption>
         </figure>
         <figure class="mdx-spotlight__feature">
-          <img
+          <img src="assets/images/spotlight/supported-registries.webp"
-            src="assets/images/spotlight/supported-registries.webp"
+               alt="OCI-compatible container registries"
-            alt="OCI-compatible container registries"
+               loading="lazy"
-            loading="lazy"
+               width="500"
-            width="500"
+               height="160"
-            height="160"
           />
           <figcaption class="md-typeset">
             <h2>Remote storage for Virtual Machines</h2>
@@ -161,39 +158,39 @@
           </figcaption>
         </figure>
         <figure class="mdx-spotlight__feature">
-          <img
+          <img src="assets/images/spotlight/github-actions-runners.webp"
-            src="assets/images/spotlight/github-actions-runners.webp"
+               alt="GitHub Actions Runners"
-            alt="GitHub Actions Runners"
+               loading="lazy"
-            loading="lazy"
+               width="500"
-            width="500"
+               height="280"
-            height="280"
           />
           <figcaption class="md-typeset">
             <h2>Seamless integration with your existing automations</h2>
             <p>
               Tart integrates with many continuous integration systems, including a dedicated
               service of on-demand GitHub Actions Runners. With a single line change, you can cut your
-              CI/CD costs by up to <b>30 times</b> by using <a href="https://cirrus-runners.app/">Cirrus Runners</a>
+              CI/CD costs by up to <b>30 times</b> by using <a href="https://cirrus-runners.app/">Cirrus
+              Runners</a>
               to run your workflows.
             </p>
           </figcaption>
         </figure>
         <figure class="mdx-spotlight__feature">
           <div id="lottie-player">
-            <dotlottie-player
+            <dotlottie-player src="/assets/animations/Orchard.lottie"
-              src="/assets/animations/Orchard.lottie"
+                              mode="normal"
-              mode="normal"
+                              style="height: 280px; margin: auto"
-              style="height: 280px; margin: auto"
+                              autoplay
-              autoplay
+                              loop
-              loop
             />
           </div>
           <figcaption class="md-typeset">
             <h2>Run at scale with <a href="https://github.com/cirruslabs/orchard">Orchard</a></h2>
             <p>
-              Tart toolset includes Orchard Orchestration &mdash; tool to run and manage Tart virtual machines
+              Tart toolset includes Orchard Orchestration &mdash; tool to run and manage Tart virtual
-              at scale on a cluster of Apple Silicon hosts. An Orchard Cluster exposes a simple REST API to manage
+              machines at scale on a cluster of Apple Silicon hosts. An Orchard Cluster exposes a simple REST API to
-              thousands virtual machines. Orchard CLI allows accessing remote virtual machines like they run locally.
+              manage thousands virtual machines. Orchard CLI allows accessing remote virtual machines like they run
+              locally.
             </p>
           </figcaption>
         </figure>
@@ -208,38 +205,38 @@
       <header class="md-typeset">
         <h1 id="powerhouse">
           Automation Powerhouse
-          <a
+          <a href="#powerhouse"
-            href="#powerhouse"
+             class="headerlink"
-            class="headerlink"
+             title="Permanent link"
-            title="Permanent link"
           >
             ¶
           </a>
         </h1>
       </header>
       <script>
-        fetch("https://api.github.com/repos/cirruslabs/tart/releases?per_page=100")
+          fetch("https://api.github.com/repos/cirruslabs/tart/releases?per_page=100")
-          .then((response) => response.json())
+              .then((response) => response.json())
-          .then((releases) => {
+              .then((releases) => {
-            let allDownloads = 0;
+                  let allDownloads = 0;
-            for (let release of releases) {
+                  for (let release of releases) {
-              for (let asset of release.assets) {
+                      for (let asset of release.assets) {
-                if (asset && asset.content_type === "application/octet-stream") {
+                          if (asset && asset.content_type === "application/octet-stream") {
-                  allDownloads += asset.download_count || 0
+                              allDownloads += asset.download_count || 0
-                }
+                          }
-              }
+                      }
-            }
+                  }
-            let counterElement = document.getElementById('installation-counter');
+                  let counterElement = document.getElementById('installation-counter');
-            if (counterElement) {
+                  if (counterElement) {
-              // Live installation count is available starting version 1.0.0
+                      // Live installation count is available starting version 1.0.0
-              // Prior Tart was installed a little over 14,000 times, let's count them too
+                      // Prior Tart was installed a little over 14,000 times, let's count them too
-              let installationPriorV1 = 14
+                      let installationPriorV1 = 14
-              counterElement.textContent = (installationPriorV1 + Math.round(allDownloads / 1000)) + ",000"
+                      counterElement.textContent = (installationPriorV1 + Math.round(allDownloads / 1000)) + ",000"
-            }
+                  }
-          })
+              })
       </script>
       <h2>
-        With more than <strong id="installation-counter">25,000</strong> installations to date, Tart has been adopted for various scenarios.
+        With more than <strong id="installation-counter">25,000</strong> installations to date, Tart has been
+        adopted for various scenarios.
         Its applications range from powering CI/CD pipelines and reproducible local development environments,
         to helping in the testing of device management systems without actual physical devices.
       </h2>
@@ -254,10 +251,9 @@
       <header class="md-typeset">
         <h1 id="what-our-users-say">
           What our users say
-          <a
+          <a href="#what-our-users-say"
-            href="#what-our-users-say"
+             class="headerlink"
-            class="headerlink"
+             title="Permanent link"
-            title="Permanent link"
           >
             ¶
           </a>
@@ -265,70 +261,60 @@
       </header>
       <div class="mdx-users">
         <figure class="mdx-users__testimonial">
-          <img
+          <img src="assets/images/users/mikhail-tokarev.webp"
-            src="assets/images/users/mitchell-hashimoto.webp"
+               alt="Mikhail Tokarev"
-            alt="Mitchell Hashimoto"
+               loading="lazy"
-            loading="lazy"
+               width="200"
-            width="200"
+               height="200"
-            height="200"
           />
           <figcaption class="md-typeset">
-            <h2>Mitchell Hashimoto</h2>
             <h3>
-              <a href="https://www.hashicorp.com/" target="_blank">HashiCorp</a> co-founder
+              Mikhail Tokarev, CTO at <a href="https://codemagic.io/start/" target="_blank">Codemagic</a>
             </h3>
             <hr/>
             <cite>
-              I've been using "Cirrus Runners" since <a href="https://x.com/mitchellh/status/1731071326201561194" target="_blank">that tweet</a> and
+              Thanks to the minimal overhead of using the Apple Virtualization
-              it has been fantastic. Huge speed increase, huge cost decrease, zero maintenance, exactly what I wanted.
+              API, we’ve seen some performance improvements in booting new
+              virtual machines compared with Anka.
             </cite>
           </figcaption>
         </figure>
         <figure class="mdx-users__testimonial">
-          <img
+          <img src="assets/images/users/expo.webp"
-            src="assets/images/users/seb-jachec.webp"
+               alt="Expo"
-            alt="Sebastian Jachec"
+               loading="lazy"
-            loading="lazy"
+               width="200"
-            width="200"
+               height="200"
-            height="200"
           />
           <figcaption class="md-typeset">
-            <h2>Sebastian Jachec</h2>
             <h3>
-              Mobile Engineer at
+              Infrastructure Team at <a href="https://expo.dev/" target="_blank">Expo</a>
-              <a href="https://daybridge.com/" target="_blank">Daybridge</a>
             </h3>
             <hr/>
             <cite>
-              It&rsquo;s been plain-sailing with the
+              Tart was the practical way for us to use the Virtualization framework. Cirrus Labs’
-              <a href="/integrations/github-actions">Cirrus Runners</a>&nbsp;&mdash;
+              continued maintenance and support gives us confidence, and it is also important for us
-              they&rsquo;ve been great! They&rsquo;re consistently&nbsp;60+%
+              to be able to read the source code when we need to understand an abstraction layer below.
-              faster on&nbsp;workflows that we&nbsp;previously used Github
-              Actions&rsquo; macOS runners for.
             </cite>
           </figcaption>
         </figure>
         <figure class="mdx-users__testimonial">
-          <img
+          <img src="assets/images/users/snowflake.webp"
-            src="assets/images/users/max-lapides.webp"
+               alt="Snowflake"
-            alt="Max Lapides"
+               loading="lazy"
-            loading="lazy"
+               width="200"
-            width="200"
+               height="200"
-            height="200"
           />
           <figcaption class="md-typeset">
-            <h2>Max Lapides</h2>
             <h3>
-              Senior Mobile Engineer at
+              Red Team at <a href="https://www.snowflake.com/" target="_blank">Snowflake</a>
-              <a href="https://www.tonal.com/" target="_blank">Tonal</a>
             </h3>
             <hr/>
             <cite>
-              Previously, we were using the GitHub&#8209;hosted macOS runners
+              The Snowflake Red Team had a need for macOS CI/CD and a segmented macOS development
-              and our iOS build took ~30&nbsp;minutes. Now with
+              environment. We solved this problem and shared our implementation with macOS EC2 and Tart.
-              <a href="/integrations/github-actions">Cirrus Runners</a>, the iOS build only
+              We also automated this process with Terraform/Packer to simplify the deployment of our
-              takes ~12&nbsp;minutes. That’s a huge boost to our productivity,
+              infrastructure and machine images.
-              and for only $150/month per runner it is much less expensive too.
             </cite>
           </figcaption>
         </figure>