Compare commits

...
33 Commits
Author SHA1 Message Date
Nikolay Edigaryev b16bbf587a Registry: log upload location when encountering blob pushing error (#121) 2022-06-08 17:23:16 +03:00
Fedor Korotkov 022317bc17 Build Debug Binary (#117)
* Build Debug Binary

Xcode 14 Beta breaks something when building with a production configuration and Tart can't run a VM.

* Fixed path
2022-06-07 18:14:52 +03:00
Fedor Korotkov 87733290e7 Smaller icon (#116)
Initially it wasn't fully following the [guidelines](https://developer.apple.com/design/human-interface-guidelines/macos/icons-and-images/app-icon/).

Fixes #112
2022-06-06 18:38:30 +03:00
Fedor Korotkov c14b46adc3 Make sure VM directory exists before moving (#111) 2022-06-03 23:22:11 +03:00
Nikolay Edigaryev 63329ef363 Atomic tart {create,clone,pull} operations using rename(2) (#109)
* tart clone: always re-generate MAC-address

This is not really an issue for non-enterprise users[1].

[1]: https://github.com/cirruslabs/tart/issues/20#issuecomment-1136944455

* Atomic tart {create,clone,pull} operations using rename(2)

* Print a nicer error message when attempting to double-run a VM

* tart clone: bring back the old MAC-address generation logic

* Ensure VMDirectory.temporary() will be deleted on failure
2022-06-01 11:02:31 -04:00
Nikolay EdigaryevandFedor Korotkov 5446164a36 tart pull: introduce --populate-cache flag (#103)
* tart pull: introduce --populate-cache flag

* VMStorageOCI: introduce cache() method

* Review comments (#107)

* Rename SetCommand back to Set

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>
2022-05-28 23:20:24 -04:00
Nikolay Edigaryev f3068b9055 tart login: verify credentials (#102)
* tart login: verify credentials

* Make DictionaryCredentialsProvider fileprivate to Login.swift
2022-05-26 09:31:46 -04:00
Fedor Korotkov afa6b7b46c Auto-detect screen DPI (#106)
* Auto-detect screen DPI

Fixes #104

* Added a comment
2022-05-26 12:23:13 +03:00
Fedor Korotkov d277fb2941 Added helpfull messages to run command (#100) 2022-05-23 11:29:35 -04:00
Fedor Korotkov 088cdc51a3 Option to run with VNC (#97)
* Option to run with VNC

So copy/paste is working. For example, `tart run --vnc latest`.

Related to 14

* Add VNC task first

* Indicate the wait
2022-05-23 16:49:15 +03:00
Fedor Korotkov afb23a3e3a Init HTTP client only if Registry is used (#99) 2022-05-21 00:26:43 +03:00
Fedor Korotkov 35904dc637 Async http client for pull/push (#95)
* Use async http client for pull/push

* Don't update progress too frequently

* Removed unused variable

* Rebased after added tests
2022-05-20 11:04:21 -04:00
Nikolay Edigaryev fec803277d Registry functional/integration tests (#96)
* Registry functional/integration tests

* Remove DockerClientSwift import

* Encodable, Decodable → Codable
2022-05-20 09:52:37 -04:00
Nikolay EdigaryevandFedor Korotkov 13b05d75c5 .ci/set-version.sh: use temporary file (#93)
* .ci/set-version.sh: use temporary file

* Update .ci/set-version.sh

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>
2022-05-19 15:51:22 -04:00
Nikolay Edigaryev 54a321df7f Sort "tart list" by default (#94) 2022-05-19 15:05:16 -04:00
Fedor Korotkov b3695c8406 Log output of hooks (#92) 2022-05-19 19:37:04 +03:00
Fedor Korotkov 0a257a1547 Option to run in recovery mode (#91)
* Option to run in recovery mode

* Fixed typo

* Use Dynamic package to call private APIs

* Improved comment

* Move options closer to invocation
2022-05-19 08:17:05 -04:00
Nikolay Edigaryev 60c15e3e49 tart list: fix heading order (#90) 2022-05-18 14:20:52 -04:00
Fedor Korotkov a1bcbdbf0b Bump default CPU cores (#86)
To prevent frustrations like #68
2022-05-18 18:12:45 +03:00
Fedor Korotkov 7fec41b2cb Document VM location on disk (#87)
Fixes #68
2022-05-18 18:01:13 +03:00
Fedor Korotkov ea4fb9a2d5 Add Geekbench report link (#83)
Fixes #82
2022-05-17 15:20:04 -04:00
Nikolay Edigaryev c2da3fd919 Support "tart --version" (#81)
* Support "tart --version"

* Move CI.swift patcher into a separate script

Otherwise it doesn't work in GoReleaser.
2022-05-17 10:40:21 -04:00
Fedor Korotkov 63a2793c32 Support upper case in remote names (#79) 2022-05-16 18:14:42 +03:00
Fedor Korotkov 0fc3d3d1f4 Add virtual sound devices (#78) 2022-05-16 17:21:21 +03:00
Nikolay Edigaryev 60b705478b Handle Ctrl + C properly (#73)
* Terminate when Ctrl+C is encountered when entering credentials

* Handle Ctrl+C by catching SIGINT and converting it to task cancellation

* maxCharacters instead of (buf.count - 2)

* readStdinCredential: fix maxCharacters to be 255

* "user" variable should be named "credential"
2022-05-16 09:56:30 -04:00
Nikolay Edigaryev fa9e3146c1 tart list: filter out uninitialized VM directories (#76) 2022-05-16 08:30:58 -04:00
Nikolay Edigaryev 3dfe8f870c Provide more details when VM's network attachment disconnects (#74) 2022-05-16 08:23:34 -04:00
Fedor Korotkov 7afa446a73 Change default disk size upon creation (#71)
To help with frustrations like https://github.com/cirruslabs/tart/issues/44#issuecomment-1126831261

Experts will still use `--disk-size` flag to tailor disk size for their needs.
2022-05-16 09:44:31 +03:00
Nikolay EdigaryevandFedor Korotkov d1bed25023 tart pull: be more liberal when accepting local image as argument (#70)
* tart pull: be more liberal when accepting local image as argument

* Update Sources/tart/Commands/Pull.swift

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>

* Single sentence and consistent capitalization

Co-authored-by: Fedor Korotkov <fedor.korotkov@gmail.com>
2022-05-15 18:03:15 -04:00
Fedor Korotkov b39c3c9b52 Reworked CI integration documentation (#69)
It was a bit confusing to folks not familiar with Cirrus CLI. Reworked the documentation to better introduce Cirrus CLI.
2022-05-15 13:25:25 +03:00
Nikolay Edigaryev 8554e56e4b Registry: use issued_at field in the token response message (#65)
* Registry: use issued_at field in the token response message

* Configure JSONDecoder instead of the Decodable itself
2022-05-12 22:33:02 +03:00
Nikolay Edigaryev 3fdcc5c33c .goreleaser(brews): depend on macOS Monterey (#64)
* .goreleaser(brews): depend on macOS Monterey

* README.md: mention that macOS Monterey or later is required
2022-05-12 00:42:12 +03:00
Austin Culter 182ddf0268 Minor typo: 'reposiotry' > 'repository' (#61)
On the tin.
2022-05-10 17:16:03 -04:00
38 changed files with 1027 additions and 310 deletions
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
TMPFILE=$(mktemp)
envsubst < Sources/tart/CI/CI.swift > $TMPFILE
mv $TMPFILE Sources/tart/CI/CI.swift
+3
View File
@@ -20,4 +20,7 @@ task:
env: env:
GITHUB_TOKEN: ENCRYPTED[!98ace8259c6024da912c14d5a3c5c6aac186890a8d4819fad78f3e0c41a4e0cd3a2537dd6e91493952fb056fa434be7c!] GITHUB_TOKEN: ENCRYPTED[!98ace8259c6024da912c14d5a3c5c6aac186890a8d4819fad78f3e0c41a4e0cd3a2537dd6e91493952fb056fa434be7c!]
GORELEASER_KEY: ENCRYPTED[!9b80b6ef684ceaf40edd4c7af93014ee156c8aba7e6e5795f41c482729887b5c31f36b651491d790f1f668670888d9fd!] GORELEASER_KEY: ENCRYPTED[!9b80b6ef684ceaf40edd4c7af93014ee156c8aba7e6e5795f41c482729887b5c31f36b651491d790f1f668670888d9fd!]
info_script:
- xcodebuild -version
- swift -version
release_script: goreleaser release_script: goreleaser
+6 -3
View File
@@ -7,12 +7,13 @@ builds:
goarch: goarch:
- arm64 - arm64
prebuilt: prebuilt:
path: .build/{{ .Arch }}-apple-macosx/release/tart path: .build/{{ .Arch }}-apple-macosx/debug/tart
before: before:
hooks: hooks:
- swift build -c release --product tart - .ci/set-version.sh
- codesign --sign - --entitlements Resources/tart.entitlements --force .build/arm64-apple-macosx/release/tart - swift build -c debug --product tart
- codesign --sign - --entitlements Resources/tart.entitlements --force .build/arm64-apple-macosx/debug/tart
archives: archives:
- id: binary - id: binary
@@ -35,3 +36,5 @@ brews:
homepage: https://github.com/cirruslabs/tart homepage: https://github.com/cirruslabs/tart
description: Run macOS VMs on Apple Silicon description: Run macOS VMs on Apple Silicon
skip_upload: auto skip_upload: auto
custom_block: |
depends_on :macos => :monterey
+72
View File
@@ -1,5 +1,23 @@
{ {
"pins" : [ "pins" : [
{
"identity" : "async-http-client",
"kind" : "remoteSourceControl",
"location" : "https://github.com/swift-server/async-http-client",
"state" : {
"revision" : "24425989dadab6d6e4167174791a23d4e2a6d0c3",
"version" : "1.10.0"
}
},
{
"identity" : "dynamic",
"kind" : "remoteSourceControl",
"location" : "https://github.com/mhdhejazi/Dynamic",
"state" : {
"branch" : "master",
"revision" : "772883073d044bc754d401cabb6574624eb3778f"
}
},
{ {
"identity" : "swift-argument-parser", "identity" : "swift-argument-parser",
"kind" : "remoteSourceControl", "kind" : "remoteSourceControl",
@@ -18,6 +36,60 @@
"version" : "0.8.1" "version" : "0.8.1"
} }
}, },
{
"identity" : "swift-log",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-log.git",
"state" : {
"revision" : "5d66f7ba25daf4f94100e7022febf3c75e37a6c7",
"version" : "1.4.2"
}
},
{
"identity" : "swift-nio",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-nio.git",
"state" : {
"revision" : "124119f0bb12384cef35aa041d7c3a686108722d",
"version" : "2.40.0"
}
},
{
"identity" : "swift-nio-extras",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-nio-extras.git",
"state" : {
"revision" : "8eea84ec6144167354387ef9244b0939f5852dc8",
"version" : "1.11.0"
}
},
{
"identity" : "swift-nio-http2",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-nio-http2.git",
"state" : {
"revision" : "72bcaf607b40d7c51044f65b0f5ed8581a911832",
"version" : "1.21.0"
}
},
{
"identity" : "swift-nio-ssl",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-nio-ssl.git",
"state" : {
"revision" : "1750873bce84b4129b5303655cce2c3d35b9ed3a",
"version" : "2.19.0"
}
},
{
"identity" : "swift-nio-transport-services",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-nio-transport-services.git",
"state" : {
"revision" : "1a4692acb88156e3da1b0c6732a8a38b2a744166",
"version" : "1.12.0"
}
},
{ {
"identity" : "swift-parsing", "identity" : "swift-parsing",
"kind" : "remoteSourceControl", "kind" : "remoteSourceControl",
+5 -1
View File
@@ -1,7 +1,6 @@
// swift-tools-version:5.6 // swift-tools-version:5.6
import PackageDescription import PackageDescription
let package = Package( let package = Package(
name: "Tart", name: "Tart",
platforms: [ platforms: [
@@ -12,13 +11,18 @@ let package = Package(
], ],
dependencies: [ dependencies: [
.package(url: "https://github.com/apple/swift-argument-parser", from: "1.1.2"), .package(url: "https://github.com/apple/swift-argument-parser", from: "1.1.2"),
.package(url: "https://github.com/mhdhejazi/Dynamic", branch: "master"),
.package(url: "https://github.com/pointfreeco/swift-parsing", from: "0.9.2"), .package(url: "https://github.com/pointfreeco/swift-parsing", from: "0.9.2"),
.package(url: "https://github.com/swift-server/async-http-client", from: "1.10.0"),
], ],
targets: [ targets: [
.executableTarget(name: "tart", dependencies: [ .executableTarget(name: "tart", dependencies: [
.product(name: "ArgumentParser", package: "swift-argument-parser"), .product(name: "ArgumentParser", package: "swift-argument-parser"),
.product(name: "AsyncHTTPClient", package: "async-http-client"),
.product(name: "Dynamic", package: "Dynamic"),
.product(name: "Parsing", package: "swift-parsing"), .product(name: "Parsing", package: "swift-parsing"),
]), ]),
.testTarget(name: "TartTests", dependencies: ["tart"]) .testTarget(name: "TartTests", dependencies: ["tart"])
] ]
) )
+18 -7
View File
@@ -3,12 +3,12 @@
*Tart* is a virtualization toolset to build, run and manage virtual machines on Apple Silicon. *Tart* is a virtualization toolset to build, run and manage virtual machines on Apple Silicon.
Built by CI engineers for your automation needs. Here are some highlights of Tart: Built by CI engineers for your automation needs. Here are some highlights of Tart:
* Tart uses Apple's own `Virtualization.Framework` for near-native performance. * Tart uses Apple's own `Virtualization.Framework` for [near-native performance](https://browser.geekbench.com/v5/cpu/compare/14966395?baseline=14966339).
* Push/Pull virtual machines from any OCI-compatible container registry. * Push/Pull virtual machines from any OCI-compatible container registry.
* Use Tart Packer Plugin to automate VM creation. * Use Tart Packer Plugin to automate VM creation.
* Built-in CI integration. * Built-in CI integration.
Try running a Tart VM on your Apple Silicon device (will download a 25 GB image): Try running a Tart VM on your Apple Silicon device running macOS Monterey or later (will download a 25 GB image):
```shell ```shell
brew install cirruslabs/cli/tart brew install cirruslabs/cli/tart
@@ -20,9 +20,13 @@ tart run monterey-base
## CI Integration ## CI Integration
[Cirrus CLI](https://github.com/cirruslabs/cirrus-cli) is an open-sourced CI-agnostic tool that can run workloads Tart itself is only responsible for managing virtual machines, but we've built Tart support into a tool called Cirrus CLI
inside containers via Docker or Podman and now inside macOS VMs via Tart. Put the following `.cirrus.yml` file also developed by Cirrus Labs. [Cirrus CLI](https://github.com/cirruslabs/cirrus-cli) is a command line tool with
in the root of your repository: one configuration format to execute common CI steps (run a script, cache a folder, etc.) locally or in any CI system.
We built Cirrus CLI to solve "But it works on my machine!" problem.
Here is an example of a `.cirrus.yml` configuration file which will start a Tart VM, will copy over working directory and
will run scripts and [other instructions](https://cirrus-ci.org/guide/writing-tasks/#supported-instructions) inside the virtual machine:
```yaml ```yaml
task: task:
@@ -37,7 +41,7 @@ task:
- sleep 15 - sleep 15
``` ```
Run it locally or in CI with the following command: Put the above `.cirrus.yml` file in the root of your repository and run it with the following command:
```shell ```shell
brew install cirruslabs/cli/cirrus brew install cirruslabs/cli/cirrus
@@ -78,7 +82,7 @@ Please refer to `tart set --help` for additional details.
### Building with Packer ### Building with Packer
Please refer to [Tart Packer Plugin reposiotry](https://github.com/cirruslabs/packer-plugin-tart) for setup instructions. Please refer to [Tart Packer Plugin repository](https://github.com/cirruslabs/packer-plugin-tart) for setup instructions.
Here is an example of a template to build `monterey-base` local image based of a remote image: Here is an example of a template to build `monterey-base` local image based of a remote image:
```json ```json
@@ -174,6 +178,13 @@ tart pull acme.io/remoteorg/name:latest my-local-vm-name
disk size for new virtual machines. Here is an example of [how to change disk size in a Packer template](https://github.com/cirruslabs/macos-image-templates/blob/fb0bcf68e0b093129136875c050205a66729b596/templates/base.pkr.hcl#L15). disk size for new virtual machines. Here is an example of [how to change disk size in a Packer template](https://github.com/cirruslabs/macos-image-templates/blob/fb0bcf68e0b093129136875c050205a66729b596/templates/base.pkr.hcl#L15).
</details> </details>
<details>
<summary>VM location on disk</summary>
Tart stores all it's files in `~/.tart/` directory. Local images that you can run are stored in `~/.tart/vms/`.
Remote images are pulled into `~/.tart/vms/cache/OCIs/`.
</details>
<details> <details>
<summary>Nested virtualization support?</summary> <summary>Nested virtualization support?</summary>
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1 version https://git-lfs.github.com/spec/v1
oid sha256:8dd6af1a08bbcdc4faf0ff53601b38136c90231e11bd81bc8cd477d6f1c7d3f2 oid sha256:1fe96aed7a965b075300f092a3ca76e09053eb7cf2f3125c3a819098a8bc4b31
size 209404 size 123360
+13
View File
@@ -0,0 +1,13 @@
struct CI {
private static let rawVersion = "${CIRRUS_TAG}"
static var version: String {
rawVersion.expanded() ? rawVersion : "SNAPSHOT"
}
}
private extension String {
func expanded() -> Bool {
!isEmpty && !starts(with: "$")
}
}
+33 -22
View File
@@ -11,24 +11,34 @@ struct Clone: AsyncParsableCommand {
@Argument(help: "new VM name") @Argument(help: "new VM name")
var newName: String var newName: String
func validate() throws {
if newName.contains("/") {
throw ValidationError("<new-name> should be a local name")
}
}
func run() async throws { func run() async throws {
do { do {
if let remoteName = try? RemoteName(sourceName) { let ociStorage = VMStorageOCI()
if !VMStorageOCI().exists(remoteName) { let localStorage = VMStorageLocal()
// Pull the VM in case it's OCI-based and doesn't exist locally yet
let registry = try Registry(host: remoteName.host, namespace: remoteName.namespace)
try await VMStorageOCI().pull(remoteName, registry: registry)
}
let remoteVM = try VMStorageHelper.open(sourceName)
let remoteConfig = try VMConfig.init(fromURL: remoteVM.configURL) if let remoteName = try? RemoteName(sourceName), !ociStorage.exists(remoteName) {
let needToGenerateNewMAC = try localVMExistsWith(macAddress: remoteConfig.macAddress.string) // Pull the VM in case it's OCI-based and doesn't exist locally yet
let registry = try Registry(host: remoteName.host, namespace: remoteName.namespace)
try remoteVM.clone(to: VMStorageLocal().create(newName), generateMAC: needToGenerateNewMAC) try await ociStorage.pull(remoteName, registry: registry)
} else {
try VMStorageHelper.open(sourceName).clone(to: VMStorageLocal().create(newName), generateMAC: true)
} }
let sourceVM = try VMStorageHelper.open(sourceName)
let generateMAC = try localStorage.hasVMsWithMACAddress(macAddress: sourceVM.macAddress())
let tmpVMDir = try VMDirectory.temporary()
try await withTaskCancellationHandler(operation: {
try sourceVM.clone(to: tmpVMDir, generateMAC: generateMAC)
try localStorage.move(newName, from: tmpVMDir)
}, onCancel: {
try? FileManager.default.removeItem(at: tmpVMDir.baseURL)
})
Foundation.exit(0) Foundation.exit(0)
} catch { } catch {
print(error) print(error)
@@ -36,15 +46,16 @@ struct Clone: AsyncParsableCommand {
Foundation.exit(1) Foundation.exit(1)
} }
} }
}
private func localVMExistsWith(macAddress: String) throws -> Bool { fileprivate extension VMDirectory {
var needToGenerateNewMAC = false func macAddress() throws -> String {
for (_, localDir) in try VMStorageLocal().list() { try VMConfig(fromURL: configURL).macAddress.string
let localConfig = try VMConfig.init(fromURL: localDir.configURL) }
if localConfig.macAddress.string == macAddress { }
needToGenerateNewMAC = true
} fileprivate extension VMStorageLocal {
} func hasVMsWithMACAddress(macAddress: String) throws -> Bool {
return needToGenerateNewMAC try list().contains { try $1.macAddress() == macAddress }
} }
} }
+12 -7
View File
@@ -13,7 +13,7 @@ struct Create: AsyncParsableCommand {
var fromIPSW: String? var fromIPSW: String?
@Option(help: ArgumentHelp("Disk size in Gb")) @Option(help: ArgumentHelp("Disk size in Gb"))
var diskSize: UInt8 = 32 var diskSize: UInt8 = 50
func validate() throws { func validate() throws {
if fromIPSW == nil { if fromIPSW == nil {
@@ -23,13 +23,18 @@ struct Create: AsyncParsableCommand {
func run() async throws { func run() async throws {
do { do {
let vmDir = try VMStorageLocal().create(name) let tmpVMDir = try VMDirectory.temporary()
try await withTaskCancellationHandler(operation: {
if fromIPSW! == "latest" {
_ = try await VM(vmDir: tmpVMDir, ipswURL: nil, diskSizeGB: diskSize)
} else {
_ = try await VM(vmDir: tmpVMDir, ipswURL: URL(fileURLWithPath: fromIPSW!), diskSizeGB: diskSize)
}
if fromIPSW! == "latest" { try VMStorageLocal().move(name, from: tmpVMDir)
_ = try await VM(vmDir: vmDir, ipswURL: nil, diskSizeGB: diskSize) }, onCancel: {
} else { try? FileManager.default.removeItem(at: tmpVMDir.baseURL)
_ = try await VM(vmDir: vmDir, ipswURL: URL(fileURLWithPath: fromIPSW!), diskSizeGB: diskSize) })
}
Foundation.exit(0) Foundation.exit(0)
} catch { } catch {
+2 -2
View File
@@ -17,7 +17,7 @@ struct IP: AsyncParsableCommand {
let vmDir = try VMStorageLocal().open(name) let vmDir = try VMStorageLocal().open(name)
let vmConfig = try VMConfig.init(fromURL: vmDir.configURL) let vmConfig = try VMConfig.init(fromURL: vmDir.configURL)
guard let ip = try await resolveIP(vmConfig, secondsToWait: wait) else { guard let ip = try await IP.resolveIP(vmConfig, secondsToWait: wait) else {
print("no IP address found, is your VM running?") print("no IP address found, is your VM running?")
Foundation.exit(1) Foundation.exit(1)
@@ -33,7 +33,7 @@ struct IP: AsyncParsableCommand {
} }
} }
private func resolveIP(_ config: VMConfig, secondsToWait: UInt16) async throws -> IPv4Address? { static public func resolveIP(_ config: VMConfig, secondsToWait: UInt16) async throws -> IPv4Address? {
let waitUntil = Calendar.current.date(byAdding: .second, value: Int(secondsToWait), to: Date.now)! let waitUntil = Calendar.current.date(byAdding: .second, value: Int(secondsToWait), to: Date.now)!
let vmMacAddress = MACAddress(fromString: config.macAddress.string)! let vmMacAddress = MACAddress(fromString: config.macAddress.string)!
+2 -2
View File
@@ -7,7 +7,7 @@ struct List: AsyncParsableCommand {
func run() async throws { func run() async throws {
do { do {
print("Name\tSource") print("Source\tName")
displayTable("local", try VMStorageLocal().list()) displayTable("local", try VMStorageLocal().list())
displayTable("oci", try VMStorageOCI().list()) displayTable("oci", try VMStorageOCI().list())
@@ -21,7 +21,7 @@ struct List: AsyncParsableCommand {
} }
private func displayTable(_ source: String, _ vms: [(String, VMDirectory)]) { private func displayTable(_ source: String, _ vms: [(String, VMDirectory)]) {
for (name, _) in vms { for (name, _) in vms.sorted(by: { left, right in left.0 < right.0 }) {
print("\(source)\t\(name)") print("\(source)\t\(name)")
} }
} }
+30 -2
View File
@@ -10,9 +10,21 @@ struct Login: AsyncParsableCommand {
func run() async throws { func run() async throws {
do { do {
let (user, password) = try Credentials.retrieveStdin() let (user, password) = try StdinCredentials.retrieve()
let credentialsProvider = DictionaryCredentialsProvider([
host: (user, password)
])
try Credentials.store(host: host, user: user, password: password) do {
let registry = try Registry(host: host, namespace: "", credentialsProvider: credentialsProvider)
try await registry.ping()
} catch {
print("invalid credentials: \(error)")
Foundation.exit(1)
}
try KeychainCredentialsProvider().store(host: host, user: user, password: password)
Foundation.exit(0) Foundation.exit(0)
} catch { } catch {
@@ -22,3 +34,19 @@ struct Login: AsyncParsableCommand {
} }
} }
} }
fileprivate class DictionaryCredentialsProvider: CredentialsProvider {
var credentials: Dictionary<String, (String, String)>
init(_ credentials: Dictionary<String, (String, String)>) {
self.credentials = credentials
}
func retrieve(host: String) throws -> (String, String)? {
credentials[host]
}
func store(host: String, user: String, password: String) throws {
credentials[host] = (user, password)
}
}
+8
View File
@@ -10,6 +10,14 @@ struct Pull: AsyncParsableCommand {
func run() async throws { func run() async throws {
do { do {
// Be more liberal when accepting local image as argument,
// see https://github.com/cirruslabs/tart/issues/36
if VMStorageLocal().exists(remoteName) {
print("\"\(remoteName)\" is a local image, nothing to pull here!")
Foundation.exit(0)
}
let remoteName = try RemoteName(remoteName) let remoteName = try RemoteName(remoteName)
let registry = try Registry(host: remoteName.host, namespace: remoteName.namespace) let registry = try Registry(host: remoteName.host, namespace: remoteName.namespace)
+28 -4
View File
@@ -12,6 +12,10 @@ struct Push: AsyncParsableCommand {
@Argument(help: "remote VM name(s)") @Argument(help: "remote VM name(s)")
var remoteNames: [String] var remoteNames: [String]
@Flag(help: ArgumentHelp("cache pushed images locally",
discussion: "Increases disk usage, but saves time if you're going to pull the pushed images later."))
var populateCache: Bool = false
func run() async throws { func run() async throws {
do { do {
let localVMDir = try VMStorageLocal().open(localName) let localVMDir = try VMStorageLocal().open(localName)
@@ -35,12 +39,20 @@ struct Push: AsyncParsableCommand {
for (registryIdentifier, remoteNamesForRegistry) in registryGroups { for (registryIdentifier, remoteNamesForRegistry) in registryGroups {
let registry = try Registry(host: registryIdentifier.host, namespace: registryIdentifier.namespace) let registry = try Registry(host: registryIdentifier.host, namespace: registryIdentifier.namespace)
let listOfTagsAndDigests = "{" + remoteNamesForRegistry.map{$0.fullyQualifiedReference }
.joined(separator: ",") + "}"
defaultLogger.appendNewLine("pushing \(localName) to " defaultLogger.appendNewLine("pushing \(localName) to "
+ "\(registryIdentifier.host)/\(registryIdentifier.namespace)\(listOfTagsAndDigests)...") + "\(registryIdentifier.host)/\(registryIdentifier.namespace)\(remoteNamesForRegistry.referenceNames())...")
try await localVMDir.pushToRegistry(registry: registry, references: remoteNamesForRegistry.map{ $0.reference }) let pushedRemoteName = try await localVMDir.pushToRegistry(registry: registry, references: remoteNamesForRegistry.map{ $0.reference.value })
// Populate the local cache (if requested)
if populateCache {
let ociStorage = VMStorageOCI()
let expectedPushedVMDir = try ociStorage.create(pushedRemoteName)
try localVMDir.clone(to: expectedPushedVMDir, generateMAC: false)
for remoteName in remoteNamesForRegistry {
try ociStorage.link(from: remoteName, to: pushedRemoteName)
}
}
} }
Foundation.exit(0) Foundation.exit(0)
@@ -51,3 +63,15 @@ struct Push: AsyncParsableCommand {
} }
} }
} }
extension Collection where Element == RemoteName {
func referenceNames() -> String {
let references = self.map{ $0.reference.fullyQualified }
switch count {
case 0: return "∅"
case 1: return references.first!
default: return "{" + references.joined(separator: ",") + "}"
}
}
}
+99 -40
View File
@@ -5,62 +5,121 @@ import Virtualization
var vm: VM? var vm: VM?
struct IPNotFound: Error {
}
struct Run: AsyncParsableCommand { struct Run: AsyncParsableCommand {
static var configuration = CommandConfiguration(abstract: "Run a VM") static var configuration = CommandConfiguration(abstract: "Run a VM")
@Argument(help: "VM name") @Argument(help: "VM name")
var name: String var name: String
@Flag var noGraphics: Bool = false @Flag(help: ArgumentHelp(
"Don't open a UI window.",
discussion: "Useful for integrating Tart VMs into other tools.\nUse `tart ip` in order to get an IP for SSHing or VNCing into the VM."))
var noGraphics: Bool = false
@Flag(help: "Boot into recovery mode")
var recovery: Bool = false
@Flag(help: ArgumentHelp(
"Use screen sharing instead of the built-in UI.",
discussion: "Useful since VNC supports copy/paste, drag and drop, etc.\nNote that Remote Login option should be enabled inside the VM."))
var vnc: Bool = false
@MainActor @MainActor
func run() async throws { func run() async throws {
if recovery && vnc {
print("You can't run in recovery and use VNC!")
Foundation.exit(1)
}
let vmDir = try VMStorageLocal().open(name) let vmDir = try VMStorageLocal().open(name)
vm = try VM(vmDir: vmDir) vm = try VM(vmDir: vmDir)
Task { await withThrowingTaskGroup(of: Void.self) { group in
do { if vnc {
try await vm!.run() group.addTask(operation: {
do {
Foundation.exit(0) print("Waiting for the VM to boot...")
} catch { let resolvedIP = try await IP.resolveIP(vm!.config, secondsToWait: 60)
print(error) guard let ip = resolvedIP else {
throw IPNotFound()
Foundation.exit(1) }
} let url = URL(string: "vnc://\(ip)")!
} print("Opening \(url)")
NSWorkspace.shared.open(url)
if noGraphics { } catch {
dispatchMain() print("Failed to get an IP for screen sharing: \(error)")
} else {
// UI mumbo-jumbo
let nsApp = NSApplication.shared
nsApp.setActivationPolicy(.regular)
nsApp.activate(ignoringOtherApps: true)
nsApp.applicationIconImage = NSImage(data: AppIconData)
struct MainApp: App {
var body: some Scene {
WindowGroup(vm!.name) {
Group {
VMView(vm: vm!).onAppear {
NSWindow.allowsAutomaticWindowTabbing = false
}
}.frame(width: CGFloat(vm!.config.display.width), height: CGFloat(vm!.config.display.height))
}.commands {
// Remove some standard menu options
CommandGroup(replacing: .help, addition: {})
CommandGroup(replacing: .newItem, addition: {})
CommandGroup(replacing: .pasteboard, addition: {})
CommandGroup(replacing: .textEditing, addition: {})
CommandGroup(replacing: .undoRedo, addition: {})
CommandGroup(replacing: .windowSize, addition: {})
} }
})
}
group.addTask {
do {
try await vm!.run(recovery)
Foundation.exit(0)
} catch {
if error.localizedDescription.contains("Failed to lock auxiliary storage.") {
print("Virtual machine \"\(name)\" is already running!")
} else {
print(error)
}
Foundation.exit(1)
} }
} }
MainApp.main() if noGraphics || vnc {
dispatchMain()
} else {
runUI()
}
}
}
private func runUI() {
let nsApp = NSApplication.shared
nsApp.setActivationPolicy(.regular)
nsApp.activate(ignoringOtherApps: true)
nsApp.applicationIconImage = NSImage(data: AppIconData)
struct MainApp: App {
var body: some Scene {
WindowGroup(vm!.name) {
Group {
VMView(vm: vm!).onAppear {
NSWindow.allowsAutomaticWindowTabbing = false
}
}.frame(width: CGFloat(vm!.config.display.width), height: CGFloat(vm!.config.display.height))
}.commands {
// Remove some standard menu options
CommandGroup(replacing: .help, addition: {})
CommandGroup(replacing: .newItem, addition: {})
CommandGroup(replacing: .pasteboard, addition: {})
CommandGroup(replacing: .textEditing, addition: {})
CommandGroup(replacing: .undoRedo, addition: {})
CommandGroup(replacing: .windowSize, addition: {})
// Replace some standard menu options
CommandGroup(replacing: .appInfo) { AboutTart() }
}
}
}
MainApp.main()
}
}
struct AboutTart: View {
var body: some View {
Button("About Tart") {
NSApplication.shared.orderFrontStandardAboutPanel(options: [
NSApplication.AboutPanelOptionKey.applicationIcon: NSApplication.shared.applicationIconImage as Any,
NSApplication.AboutPanelOptionKey.applicationName: "Tart",
NSApplication.AboutPanelOptionKey.applicationVersion: CI.version,
NSApplication.AboutPanelOptionKey.credits: try! NSAttributedString(markdown: "https://github.com/cirruslabs/tart"),
])
} }
} }
} }
+3 -7
View File
@@ -2,7 +2,7 @@ import ArgumentParser
import Foundation import Foundation
struct Set: AsyncParsableCommand { struct Set: AsyncParsableCommand {
static var configuration = CommandConfiguration(abstract: "Modify VM's configuration") static var configuration = CommandConfiguration(commandName: "set", abstract: "Modify VM's configuration")
@Argument(help: "VM name") @Argument(help: "VM name")
var name: String var name: String
@@ -13,7 +13,7 @@ struct Set: AsyncParsableCommand {
@Option(help: "VM memory size in megabytes") @Option(help: "VM memory size in megabytes")
var memory: UInt16? var memory: UInt16?
@Option(help: "VM display settings in a format of <width>x<height>(x<dpi>)?. For example, 1200x800 or 1200x800x72") @Option(help: "VM display resolution in a format of <width>x<height>. For example, 1200x800")
var display: VMDisplayConfig? var display: VMDisplayConfig?
@Option(help: .hidden) @Option(help: .hidden)
@@ -39,9 +39,6 @@ struct Set: AsyncParsableCommand {
if (display.height > 0) { if (display.height > 0) {
vmConfig.display.height = display.height vmConfig.display.height = display.height
} }
if (display.dpi > 0) {
vmConfig.display.dpi = display.dpi
}
} }
try vmConfig.save(toURL: vmDir.configURL) try vmConfig.save(toURL: vmDir.configURL)
@@ -66,8 +63,7 @@ extension VMDisplayConfig: ExpressibleByArgument {
} }
self = VMDisplayConfig( self = VMDisplayConfig(
width: parts[safe: 0] ?? 0, width: parts[safe: 0] ?? 0,
height: parts[safe: 1] ?? 0, height: parts[safe: 1] ?? 0
dpi: parts[safe: 2] ?? 0
) )
} }
} }
-64
View File
@@ -1,64 +0,0 @@
import Foundation
class Credentials {
static func retrieveKeychain(host: String) throws -> (String, String)? {
let query: [String: Any] = [kSecClass as String: kSecClassInternetPassword,
kSecAttrProtocol as String: kSecAttrProtocolHTTPS,
kSecAttrServer as String: host,
kSecMatchLimit as String: kSecMatchLimitOne,
kSecReturnAttributes as String: true,
kSecReturnData as String: true,
kSecAttrLabel as String: "Tart Credentials",
]
var item: CFTypeRef?
let status = SecItemCopyMatching(query as CFDictionary, &item)
if status != errSecSuccess {
if status == errSecItemNotFound {
return nil
}
throw RegistryError.AuthFailed(why: "Keychain returned unsuccessful status \(status)")
}
guard let item = item as? [String: Any],
let user = item[kSecAttrAccount as String] as? String,
let passwordData = item[kSecValueData as String] as? Data,
let password = String(data: passwordData, encoding: .utf8)
else {
throw RegistryError.AuthFailed(why: "Keychain item has unexpected format")
}
return (user, password)
}
static func retrieveStdin() throws -> (String, String) {
print("User: ", terminator: "")
let user = readLine() ?? ""
let rawPass = getpass("Password: ")
let pass = String(cString: rawPass!, encoding: .utf8)!
return (user, pass)
}
static func store(host: String, user: String, password: String) throws {
let attributes: [String: Any] = [kSecClass as String: kSecClassInternetPassword,
kSecAttrAccount as String: user,
kSecAttrProtocol as String: kSecAttrProtocolHTTPS,
kSecAttrServer as String: host,
kSecValueData as String: password,
kSecAttrLabel as String: "Tart Credentials",
]
let status = SecItemAdd(attributes as CFDictionary, nil)
switch status {
case errSecSuccess, errSecDuplicateItem:
return
default:
throw RegistryError.AuthFailed(why: "Keychain returned unsuccessful status \(status)")
}
}
}
@@ -0,0 +1,10 @@
import Foundation
enum CredentialsProviderError: Error {
case Failed(message: String)
}
protocol CredentialsProvider {
func retrieve(host: String) throws -> (String, String)?
func store(host: String, user: String, password: String) throws
}
@@ -0,0 +1,54 @@
import Foundation
class KeychainCredentialsProvider: CredentialsProvider {
func retrieve(host: String) throws -> (String, String)? {
let query: [String: Any] = [kSecClass as String: kSecClassInternetPassword,
kSecAttrProtocol as String: kSecAttrProtocolHTTPS,
kSecAttrServer as String: host,
kSecMatchLimit as String: kSecMatchLimitOne,
kSecReturnAttributes as String: true,
kSecReturnData as String: true,
kSecAttrLabel as String: "Tart Credentials",
]
var item: CFTypeRef?
let status = SecItemCopyMatching(query as CFDictionary, &item)
if status != errSecSuccess {
if status == errSecItemNotFound {
return nil
}
throw CredentialsProviderError.Failed(message: "Keychain returned unsuccessful status \(status)")
}
guard let item = item as? [String: Any],
let user = item[kSecAttrAccount as String] as? String,
let passwordData = item[kSecValueData as String] as? Data,
let password = String(data: passwordData, encoding: .utf8)
else {
throw CredentialsProviderError.Failed(message: "Keychain item has unexpected format")
}
return (user, password)
}
func store(host: String, user: String, password: String) throws {
let attributes: [String: Any] = [kSecClass as String: kSecClassInternetPassword,
kSecAttrAccount as String: user,
kSecAttrProtocol as String: kSecAttrProtocolHTTPS,
kSecAttrServer as String: host,
kSecValueData as String: password,
kSecAttrLabel as String: "Tart Credentials",
]
let status = SecItemAdd(attributes as CFDictionary, nil)
switch status {
case errSecSuccess, errSecDuplicateItem:
return
default:
throw CredentialsProviderError.Failed(message: "Keychain returned unsuccessful status \(status)")
}
}
}
@@ -0,0 +1,31 @@
import Foundation
enum StdinCredentialsError: Error {
case CredentialRequired(which: String)
case CredentialTooLong(message: String)
}
class StdinCredentials {
static func retrieve() throws -> (String, String) {
let user = try readStdinCredential(name: "username", prompt: "User: ", isSensitive: false)
let password = try readStdinCredential(name: "password", prompt: "Password: ", isSensitive: true)
return (user, password)
}
private static func readStdinCredential(name: String, prompt: String, maxCharacters: Int = 255, isSensitive: Bool) throws -> String {
var buf = [CChar](repeating: 0, count: maxCharacters + 1 /* sentinel */ + 1 /* NUL */)
guard let rawCredential = readpassphrase(prompt, &buf, buf.count, isSensitive ? RPP_ECHO_OFF : RPP_ECHO_ON) else {
throw StdinCredentialsError.CredentialRequired(which: name)
}
let credential = String(cString: rawCredential).trimmingCharacters(in: .newlines)
if credential.count > maxCharacters {
throw StdinCredentialsError.CredentialTooLong(
message: "\(name) should contain no more than \(maxCharacters) characters")
}
return credential
}
}
File diff suppressed because one or more lines are too long
+6 -1
View File
@@ -3,6 +3,7 @@ import Foundation
public class ProgressObserver: NSObject { public class ProgressObserver: NSObject {
@objc var progressToObserve: Progress @objc var progressToObserve: Progress
var observation: NSKeyValueObservation? var observation: NSKeyValueObservation?
var lastTimeUpdated = Date.now
public init(_ progress: Progress) { public init(_ progress: Progress) {
progressToObserve = progress progressToObserve = progress
@@ -11,7 +12,11 @@ public class ProgressObserver: NSObject {
func log(_ renderer: Logger) { func log(_ renderer: Logger) {
renderer.appendNewLine(ProgressObserver.lineToRender(progressToObserve)) renderer.appendNewLine(ProgressObserver.lineToRender(progressToObserve))
observation = observe(\.progressToObserve.fractionCompleted) { progress, _ in observation = observe(\.progressToObserve.fractionCompleted) { progress, _ in
renderer.updateLastLine(ProgressObserver.lineToRender(self.progressToObserve)) let currentTime = Date.now
if self.progressToObserve.isFinished || currentTime.timeIntervalSince(self.lastTimeUpdated) >= 1.0 {
self.lastTimeUpdated = currentTime
renderer.updateLastLine(ProgressObserver.lineToRender(self.progressToObserve))
}
} }
} }
+8 -4
View File
@@ -3,26 +3,30 @@ import Foundation
let ociManifestMediaType = "application/vnd.oci.image.manifest.v1+json" let ociManifestMediaType = "application/vnd.oci.image.manifest.v1+json"
let ociConfigMediaType = "application/vnd.oci.image.config.v1+json" let ociConfigMediaType = "application/vnd.oci.image.config.v1+json"
struct OCIManifest: Encodable, Decodable { struct OCIManifest: Codable, Equatable {
var schemaVersion: Int = 2 var schemaVersion: Int = 2
var mediaType: String = ociManifestMediaType var mediaType: String = ociManifestMediaType
var config: OCIManifestConfig var config: OCIManifestConfig
var layers: [OCIManifestLayer] = Array() var layers: [OCIManifestLayer] = Array()
func digest() throws -> String {
try Digest.hash(JSONEncoder().encode(self))
}
} }
struct OCIManifestConfig: Encodable, Decodable { struct OCIManifestConfig: Codable, Equatable {
var mediaType: String = ociConfigMediaType var mediaType: String = ociConfigMediaType
var size: Int var size: Int
var digest: String var digest: String
} }
struct OCIManifestLayer: Encodable, Decodable { struct OCIManifestLayer: Codable, Equatable {
var mediaType: String var mediaType: String
var size: Int var size: Int
var digest: String var digest: String
} }
struct Descriptor { struct Descriptor: Equatable {
var size: Int var size: Int
var digest: String var digest: String
} }
+140 -68
View File
@@ -1,18 +1,56 @@
import Foundation import Foundation
import NIOCore
import NIOHTTP1
import AsyncHTTPClient
enum RegistryError: Error { enum RegistryError: Error {
case UnexpectedHTTPStatusCode(when: String, code: Int, details: String = "") case UnexpectedHTTPStatusCode(when: String, code: UInt, details: String = "")
case MissingLocationHeader case MissingLocationHeader
case AuthFailed(why: String, details: String = "") case AuthFailed(why: String, details: String = "")
case MalformedHeader(why: String) case MalformedHeader(why: String)
} }
extension HTTPClientResponse.Body {
func readTextResponse() async throws -> String? {
let data = try await readResponse()
return String(decoding: data, as: UTF8.self)
}
func readResponse() async throws -> Data {
var result = Data()
for try await part in self {
result.append(Data(buffer: part))
}
return result
}
}
struct TokenResponse: Decodable { struct TokenResponse: Decodable {
let creationTime = Date() let defaultIssuedAt = Date()
let defaultExpiresIn = 60
var token: String var token: String
var expires_in: Int? var expiresIn: Int?
var issuedAt: Date?
static func parse(fromData: Data) throws -> Self {
let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase
// RFC3339 date formatter from Apple's documentation[1]
//
// [1]: https://developer.apple.com/documentation/foundation/dateformatter
let dateFormatter = DateFormatter()
dateFormatter.locale = Locale(identifier: "en_US_POSIX")
dateFormatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ssZZZZZ"
dateFormatter.timeZone = TimeZone(secondsFromGMT: 0)
decoder.dateDecodingStrategy = .formatted(dateFormatter)
return try decoder.decode(TokenResponse.self, from: fromData)
}
var tokenExpiresAt: Date { var tokenExpiresAt: Date {
get { get {
// Tokens can expire and expire_in field is used to determine when: // Tokens can expire and expire_in field is used to determine when:
@@ -22,11 +60,11 @@ struct TokenResponse: Decodable {
// >a token should never be returned with less than 60 seconds to live. // >a token should never be returned with less than 60 seconds to live.
// //
// [1]: https://docs.docker.com/registry/spec/auth/token/#requesting-a-token // [1]: https://docs.docker.com/registry/spec/auth/token/#requesting-a-token
creationTime + TimeInterval(expires_in ?? 60) (issuedAt ?? defaultIssuedAt) + TimeInterval(expiresIn ?? defaultExpiresIn)
} }
} }
var isValid: Bool { var isValid: Bool {
get { get {
Date() < tokenExpiresAt Date() < tokenExpiresAt
@@ -35,52 +73,79 @@ struct TokenResponse: Decodable {
} }
class Registry { class Registry {
var baseURL: URL private let httpClient = HTTPClient(eventLoopGroupProvider: .createNew)
var namespace: String
deinit {
try! httpClient.syncShutdown()
}
let baseURL: URL
let namespace: String
let credentialsProvider: CredentialsProvider
var currentAuthToken: TokenResponse? = nil var currentAuthToken: TokenResponse? = nil
init(host: String, namespace: String) throws { init(urlComponents: URLComponents,
namespace: String,
credentialsProvider: CredentialsProvider = KeychainCredentialsProvider()
) throws {
baseURL = urlComponents.url!
self.namespace = namespace
self.credentialsProvider = credentialsProvider
}
convenience init(
host: String,
namespace: String,
credentialsProvider: CredentialsProvider = KeychainCredentialsProvider()
) throws {
var baseURLComponents = URLComponents() var baseURLComponents = URLComponents()
baseURLComponents.scheme = "https" baseURLComponents.scheme = "https"
baseURLComponents.host = host baseURLComponents.host = host
baseURLComponents.path = "/v2/" baseURLComponents.path = "/v2/"
baseURL = baseURLComponents.url! try self.init(urlComponents: baseURLComponents, namespace: namespace, credentialsProvider: credentialsProvider)
self.namespace = namespace
} }
func pushManifest(reference: String, config: Descriptor, layers: [OCIManifestLayer]) async throws -> String { func ping() async throws {
let manifest = OCIManifest(config: OCIManifestConfig(size: config.size, digest: config.digest), let response = try await endpointRequest(.GET, "/v2/")
layers: layers) if response.status != .ok {
throw RegistryError.UnexpectedHTTPStatusCode(when: "doing ping", code: response.status.code)
}
}
func pushManifest(reference: String, manifest: OCIManifest) async throws -> String {
let manifestJSON = try JSONEncoder().encode(manifest) let manifestJSON = try JSONEncoder().encode(manifest)
let (responseData, response) = try await endpointRequest("PUT", "\(namespace)/manifests/\(reference)", let response = try await endpointRequest(.PUT, "\(namespace)/manifests/\(reference)",
headers: ["Content-Type": manifest.mediaType], headers: ["Content-Type": manifest.mediaType],
body: manifestJSON) body: manifestJSON)
if response.statusCode != 201 { if response.status != .created {
throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing manifest", code: response.statusCode, throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing manifest", code: response.status.code,
details: String(decoding: responseData, as: UTF8.self)) details: try await response.body.readTextResponse() ?? "")
} }
return Digest.hash(manifestJSON) return Digest.hash(manifestJSON)
} }
public func pullManifest(reference: String) async throws -> (OCIManifest, Data) { public func pullManifest(reference: String) async throws -> (OCIManifest, Data) {
let (responseData, response) = try await endpointRequest("GET", "\(namespace)/manifests/\(reference)", let response = try await endpointRequest(.GET, "\(namespace)/manifests/\(reference)",
headers: ["Accept": ociManifestMediaType]) headers: ["Accept": ociManifestMediaType])
if response.statusCode != 200 { if response.status != .ok {
throw RegistryError.UnexpectedHTTPStatusCode(when: "pulling manifest", code: response.statusCode, let body = try await response.body.readTextResponse()
details: String(decoding: responseData, as: UTF8.self)) throw RegistryError.UnexpectedHTTPStatusCode(when: "pulling manifest", code: response.status.code,
details: body ?? "")
} }
let manifest = try JSONDecoder().decode(OCIManifest.self, from: responseData) let manifestData = try await response.body.readResponse()
let manifest = try JSONDecoder().decode(OCIManifest.self, from: manifestData)
return (manifest, responseData) return (manifest, manifestData)
} }
private func uploadLocationFromResponse(response: HTTPURLResponse) throws -> URLComponents { private func uploadLocationFromResponse(_ response: HTTPClientResponse) throws -> URLComponents {
guard let uploadLocationRaw = response.value(forHTTPHeaderField: "Location") else { guard let uploadLocationRaw = response.headers.first(name: "Location") else {
throw RegistryError.MissingLocationHeader throw RegistryError.MissingLocationHeader
} }
@@ -93,15 +158,16 @@ class Registry {
public func pushBlob(fromData: Data, chunkSize: Int = 5 * 1024 * 1024) async throws -> String { public func pushBlob(fromData: Data, chunkSize: Int = 5 * 1024 * 1024) async throws -> String {
// Initiate a blob upload // Initiate a blob upload
let (postData, postResponse) = try await endpointRequest("POST", "\(namespace)/blobs/uploads/", let postResponse = try await endpointRequest(.POST, "\(namespace)/blobs/uploads/",
headers: ["Content-Length": "0"]) headers: ["Content-Length": "0"])
if postResponse.statusCode != 202 { if postResponse.status != .accepted {
throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing blob (POST)", code: postResponse.statusCode, let body = try await postResponse.body.readTextResponse()
details: String(decoding: postData, as: UTF8.self)) throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing blob (POST)", code: postResponse.status.code,
details: body ?? "")
} }
// Figure out where to upload the blob // Figure out where to upload the blob
let uploadLocation = try uploadLocationFromResponse(response: postResponse) let uploadLocation = try uploadLocationFromResponse(postResponse)
// Upload the blob // Upload the blob
let headers = [ let headers = [
@@ -114,33 +180,37 @@ class Registry {
"digest": digest, "digest": digest,
] ]
let (putData, putResponse) = try await rawRequest("PUT", uploadLocation, headers: headers, parameters: parameters, let putResponse = try await rawRequest(.PUT, uploadLocation, headers: headers, parameters: parameters,
body: fromData) body: fromData)
if putResponse.statusCode != 201 { if putResponse.status != .created {
throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing blob (PUT)", code: putResponse.statusCode, let body = try await postResponse.body.readTextResponse()
details: String(decoding: putData, as: UTF8.self)) throw RegistryError.UnexpectedHTTPStatusCode(when: "pushing blob (PUT) to \(uploadLocation)",
code: putResponse.status.code, details: body ?? "")
} }
return digest return digest
} }
public func pullBlob(_ digest: String) async throws -> Data { public func pullBlob(_ digest: String, handler: (ByteBuffer) throws -> Void) async throws {
let (putData, putResponse) = try await endpointRequest("GET", "\(namespace)/blobs/\(digest)") let response = try await endpointRequest(.GET, "\(namespace)/blobs/\(digest)")
if putResponse.statusCode != 200 { if response.status != .ok {
throw RegistryError.UnexpectedHTTPStatusCode(when: "pulling blob", code: putResponse.statusCode, let body = try await response.body.readTextResponse()
details: String(decoding: putData, as: UTF8.self)) throw RegistryError.UnexpectedHTTPStatusCode(when: "pulling blob", code: response.status.code,
details: body ?? "")
} }
return putData for try await part in response.body {
try handler(part)
}
} }
private func endpointRequest( private func endpointRequest(
_ method: String, _ method: HTTPMethod,
_ endpoint: String, _ endpoint: String,
headers: Dictionary<String, String> = Dictionary(), headers: Dictionary<String, String> = Dictionary(),
parameters: Dictionary<String, String> = Dictionary(), parameters: Dictionary<String, String> = Dictionary(),
body: Data? = nil body: Data? = nil
) async throws -> (Data, HTTPURLResponse) { ) async throws -> HTTPClientResponse {
let url = URL(string: endpoint, relativeTo: baseURL)! let url = URL(string: endpoint, relativeTo: baseURL)!
let urlComponents = URLComponents(url: url, resolvingAgainstBaseURL: true)! let urlComponents = URLComponents(url: url, resolvingAgainstBaseURL: true)!
@@ -148,12 +218,12 @@ class Registry {
} }
private func rawRequest( private func rawRequest(
_ method: String, _ method: HTTPMethod,
_ urlComponents: URLComponents, _ urlComponents: URLComponents,
headers: Dictionary<String, String> = Dictionary(), headers: Dictionary<String, String> = Dictionary(),
parameters: Dictionary<String, String> = Dictionary(), parameters: Dictionary<String, String> = Dictionary(),
body: Data? = nil body: Data? = nil
) async throws -> (Data, HTTPURLResponse) { ) async throws -> HTTPClientResponse {
var urlComponents = urlComponents var urlComponents = urlComponents
if urlComponents.queryItems == nil { if urlComponents.queryItems == nil {
@@ -163,31 +233,33 @@ class Registry {
URLQueryItem(name: key, value: value) URLQueryItem(name: key, value: value)
}) })
var request = URLRequest(url: urlComponents.url!) var request = HTTPClientRequest(url: urlComponents.string!)
request.httpMethod = method request.method = method
for (key, value) in headers { for (key, value) in headers {
request.addValue(value, forHTTPHeaderField: key) request.headers.add(name: key, value: value)
}
if body != nil {
request.body = HTTPClientRequest.Body.bytes(body!)
} }
request.httpBody = body
// Invalidate token if it has expired // Invalidate token if it has expired
if currentAuthToken?.isValid == false { if currentAuthToken?.isValid == false {
currentAuthToken = nil currentAuthToken = nil
} }
var (data, response) = try await authAwareRequest(request: request) var response = try await authAwareRequest(request: request)
if response.statusCode == 401 { if response.status == .unauthorized {
try await auth(response: response) try await auth(response: response)
(data, response) = try await authAwareRequest(request: request) response = try await authAwareRequest(request: request)
} }
return (data, response) return response
} }
private func auth(response: HTTPURLResponse) async throws { private func auth(response: HTTPClientResponse) async throws {
// Process WWW-Authenticate header // Process WWW-Authenticate header
guard let wwwAuthenticateRaw = response.value(forHTTPHeaderField: "WWW-Authenticate") else { guard let wwwAuthenticateRaw = response.headers.first(name: "WWW-Authenticate") else {
throw RegistryError.AuthFailed(why: "got HTTP 401, but WWW-Authenticate header is missing") throw RegistryError.AuthFailed(why: "got HTTP 401, but WWW-Authenticate header is missing")
} }
@@ -222,29 +294,29 @@ class Registry {
var headers: Dictionary<String, String> = Dictionary() var headers: Dictionary<String, String> = Dictionary()
if let (user, password) = try Credentials.retrieveKeychain(host: baseURL.host!) { if let (user, password) = try credentialsProvider.retrieve(host: baseURL.host!) {
let encodedCredentials = "\(user):\(password)".data(using: .utf8)?.base64EncodedString() let encodedCredentials = "\(user):\(password)".data(using: .utf8)?.base64EncodedString()
headers["Authorization"] = "Basic \(encodedCredentials!)" headers["Authorization"] = "Basic \(encodedCredentials!)"
} }
let (tokenResponseRaw, response) = try await rawRequest("GET", authenticateURL, headers: headers) let response = try await rawRequest(.GET, authenticateURL, headers: headers)
if response.statusCode != 200 { if response.status != .ok {
throw RegistryError.AuthFailed(why: "received unexpected HTTP status code \(response.statusCode) " let body = try await response.body.readTextResponse() ?? ""
+ "while retrieving an authentication token", details: String(decoding: tokenResponseRaw, as: UTF8.self)) throw RegistryError.AuthFailed(why: "received unexpected HTTP status code \(response.status.code) "
+ "while retrieving an authentication token", details: body)
} }
currentAuthToken = try JSONDecoder().decode(TokenResponse.self, from: tokenResponseRaw) let bodyData = try await response.body.readResponse()
currentAuthToken = try TokenResponse.parse(fromData: bodyData)
} }
private func authAwareRequest(request: URLRequest) async throws -> (Data, HTTPURLResponse) { private func authAwareRequest(request: HTTPClientRequest) async throws -> HTTPClientResponse {
var request = request var request = request
if let token = currentAuthToken { if let token = currentAuthToken {
request.addValue("Bearer \(token.token)", forHTTPHeaderField: "Authorization") request.headers.add(name: "Authorization", value: "Bearer \(token.token)")
} }
let (responseData, response) = try await URLSession.shared.data(for: request) return try await httpClient.execute(request, deadline: .distantFuture)
return (responseData, response as! HTTPURLResponse)
} }
} }
+46 -21
View File
@@ -1,31 +1,57 @@
import Foundation import Foundation
import Parsing import Parsing
struct Tail { struct Reference: Comparable, Hashable, CustomStringConvertible {
enum TailType { enum ReferenceType: Comparable {
case Tag case Tag
case Digest case Digest
} }
var type: TailType let type: ReferenceType
var value: String let value: String
}
struct RemoteName: Comparable, CustomStringConvertible { var fullyQualified: String {
var host: String
var namespace: String
var reference: String = "latest"
var fullyQualifiedReference: String {
get { get {
if reference.starts(with: "sha256:") { switch type {
return "@" + reference case .Tag:
return ":" + value
case .Digest:
return "@" + value
} }
return ":" + reference
} }
} }
init(host: String, namespace: String, reference: String) { init(tag: String) {
type = .Tag
value = tag
}
init(digest: String) {
type = .Digest
value = digest
}
static func <(lhs: Reference, rhs: Reference) -> Bool {
if lhs.type != rhs.type {
return lhs.type < rhs.type
} else {
return lhs.value < rhs.value
}
}
var description: String {
get {
fullyQualified
}
}
}
struct RemoteName: Comparable, Hashable, CustomStringConvertible {
var host: String
var namespace: String
var reference: Reference
init(host: String, namespace: String, reference: Reference) {
self.host = host self.host = host
self.namespace = namespace self.namespace = namespace
self.reference = reference self.reference = reference
@@ -34,6 +60,7 @@ struct RemoteName: Comparable, CustomStringConvertible {
init(_ name: String) throws { init(_ name: String) throws {
let csNormal = [ let csNormal = [
UInt8(ascii: "a")...UInt8(ascii: "z"), UInt8(ascii: "a")...UInt8(ascii: "z"),
UInt8(ascii: "A")...UInt8(ascii: "Z"),
UInt8(ascii: "0")...UInt8(ascii: "9"), UInt8(ascii: "0")...UInt8(ascii: "9"),
].asCharacterSet().union(CharacterSet(charactersIn: "_-.")) ].asCharacterSet().union(CharacterSet(charactersIn: "_-."))
@@ -57,13 +84,13 @@ struct RemoteName: Comparable, CustomStringConvertible {
Parse { Parse {
":" ":"
csNormal.map { csNormal.map {
Tail(type: .Tag, value: String($0)) Reference(tag: String($0))
} }
} }
Parse { Parse {
"@sha256:" "@sha256:"
csHex.map { csHex.map {
Tail(type: .Digest, value: "sha256:" + String($0)) Reference(digest: "sha256:" + String($0))
} }
} }
} }
@@ -75,9 +102,7 @@ struct RemoteName: Comparable, CustomStringConvertible {
host = String(result.0) host = String(result.0)
namespace = String(result.1) namespace = String(result.1)
if let tail = result.2 { reference = result.2 ?? Reference(tag: "latest")
reference = tail.value
}
} }
static func <(lhs: RemoteName, rhs: RemoteName) -> Bool { static func <(lhs: RemoteName, rhs: RemoteName) -> Bool {
@@ -91,7 +116,7 @@ struct RemoteName: Comparable, CustomStringConvertible {
} }
var description: String { var description: String {
"\(host)/\(namespace)\(fullyQualifiedReference)" "\(host)/\(namespace)\(reference.fullyQualified)"
} }
} }
+29
View File
@@ -1,9 +1,11 @@
import ArgumentParser import ArgumentParser
import Foundation
@main @main
struct Root: AsyncParsableCommand { struct Root: AsyncParsableCommand {
static var configuration = CommandConfiguration( static var configuration = CommandConfiguration(
commandName: "tart", commandName: "tart",
version: CI.version,
subcommands: [ subcommands: [
Create.self, Create.self,
Clone.self, Clone.self,
@@ -16,4 +18,31 @@ struct Root: AsyncParsableCommand {
Push.self, Push.self,
Delete.self, Delete.self,
]) ])
public static func main() async throws {
// Ensure the default SIGINT handled is disabled,
// otherwise there's a race between two handlers
signal(SIGINT, SIG_IGN);
// Handle cancellation by Ctrl+C ourselves
let task = withUnsafeCurrentTask { $0 }!
let sigintSrc = DispatchSource.makeSignalSource(signal: SIGINT)
sigintSrc.setEventHandler {
task.cancel()
Darwin.exit(1)
}
sigintSrc.activate()
// Parse and run command
do {
var command = try parseAsRoot()
if var asyncCommand = command as? AsyncParsableCommand {
try await asyncCommand.run()
} else {
try command.run()
}
} catch {
exit(withError: error)
}
}
} }
+37
View File
@@ -0,0 +1,37 @@
import Foundation
import Virtualization
import Dynamic
// Kudos to @saagarjha's VirtualApple for finding about _VZVirtualMachineStartOptions
extension VZVirtualMachine {
func start(_ recovery: Bool) async throws {
if !recovery {
// just use the regular API
return try await withCheckedThrowingContinuation { continuation in
DispatchQueue.main.async {
self.start(completionHandler: { result in
continuation.resume(with: result)
})
}
}
}
// use some private stuff only for recovery
return try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
DispatchQueue.main.async {
let handler: @convention(block) (_ result: Any?) -> Void = { result in
if let error = result as? Error {
continuation.resume(throwing: error)
} else {
continuation.resume(returning: ())
}
}
// dynamic magic
let options = Dynamic._VZVirtualMachineStartOptions()
options.bootMacOSRecovery = recovery
Dynamic(self)._start(withOptions: options, completionHandler: handler)
}
}
}
}
+39 -15
View File
@@ -106,6 +106,8 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
cpuCountMin: requirements.minimumSupportedCPUCount, cpuCountMin: requirements.minimumSupportedCPUCount,
memorySizeMin: requirements.minimumSupportedMemorySize memorySizeMin: requirements.minimumSupportedMemorySize
) )
// allocate at least 4 CPUs because otherwise VMs are frequently freezing
try config.setCPU(cpuCount: max(4, requirements.minimumSupportedCPUCount))
try config.save(toURL: vmDir.configURL) try config.save(toURL: vmDir.configURL)
// Initialize the virtual machine and its configuration // Initialize the virtual machine and its configuration
@@ -131,16 +133,22 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
} }
} }
func run() async throws { func run(_ recovery: Bool) async throws {
try await withCheckedThrowingContinuation { continuation in try await virtualMachine.start(recovery)
DispatchQueue.main.async {
self.virtualMachine.start(completionHandler: { result in await withTaskCancellationHandler(operation: {
continuation.resume(with: result) sema.wait()
}) }, onCancel: {
sema.signal()
})
if Task.isCancelled {
DispatchQueue.main.sync {
Task {
try await self.virtualMachine.stop()
}
} }
} }
sema.wait()
} }
static func craftConfiguration(diskURL: URL, auxStorage: VZMacAuxiliaryStorage, vmConfig: VMConfig) throws -> VZVirtualMachineConfiguration { static func craftConfiguration(diskURL: URL, auxStorage: VZMacAuxiliaryStorage, vmConfig: VMConfig) throws -> VZVirtualMachineConfiguration {
@@ -164,15 +172,31 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
// Display // Display
let graphicsDeviceConfiguration = VZMacGraphicsDeviceConfiguration() let graphicsDeviceConfiguration = VZMacGraphicsDeviceConfiguration()
graphicsDeviceConfiguration.displays = [ if let hostMainScreen = NSScreen.main {
VZMacGraphicsDisplayConfiguration( let vmScreenSize = NSSize(
widthInPixels: vmConfig.display.width, width: vmConfig.display.width,
heightInPixels: vmConfig.display.height, height: vmConfig.display.height
pixelsPerInch: vmConfig.display.dpi
) )
] graphicsDeviceConfiguration.displays = [
VZMacGraphicsDisplayConfiguration(for: hostMainScreen, sizeInPoints: vmScreenSize)
]
} else {
graphicsDeviceConfiguration.displays = [
VZMacGraphicsDisplayConfiguration(
widthInPixels: vmConfig.display.width,
heightInPixels: vmConfig.display.height,
// Reasonable guess like https://developer.apple.com/documentation/coregraphics/1456599-cgdisplayscreensize
pixelsPerInch: 72
)
]
}
configuration.graphicsDevices = [graphicsDeviceConfiguration] configuration.graphicsDevices = [graphicsDeviceConfiguration]
// Audio
let soundDeviceConfiguration = VZVirtioSoundDeviceConfiguration()
soundDeviceConfiguration.streams = [VZVirtioSoundDeviceInputStreamConfiguration(), VZVirtioSoundDeviceOutputStreamConfiguration()]
configuration.audioDevices = [soundDeviceConfiguration]
// Keyboard and mouse // Keyboard and mouse
configuration.keyboards = [VZUSBKeyboardConfiguration()] configuration.keyboards = [VZUSBKeyboardConfiguration()]
configuration.pointingDevices = [VZUSBScreenCoordinatePointingDeviceConfiguration()] configuration.pointingDevices = [VZUSBScreenCoordinatePointingDeviceConfiguration()]
@@ -207,7 +231,7 @@ class VM: NSObject, VZVirtualMachineDelegate, ObservableObject {
} }
func virtualMachine(_ virtualMachine: VZVirtualMachine, networkDevice: VZNetworkDevice, attachmentWasDisconnectedWithError error: Error) { func virtualMachine(_ virtualMachine: VZVirtualMachine, networkDevice: VZNetworkDevice, attachmentWasDisconnectedWithError error: Error) {
print("virtual machine's network attachment has been disconnected") print("virtual machine's network attachment \(networkDevice) has been disconnected with error: \(error)")
sema.signal() sema.signal()
} }
} }
-1
View File
@@ -29,7 +29,6 @@ enum CodingKeys: String, CodingKey {
struct VMDisplayConfig: Codable { struct VMDisplayConfig: Codable {
var width: Int = 1024 var width: Int = 1024
var height: Int = 768 var height: Int = 768
var dpi: Int = 72
} }
struct VMConfig: Codable { struct VMConfig: Codable {
+41 -18
View File
@@ -4,7 +4,7 @@ import Compression
enum OCIError: Error { enum OCIError: Error {
case ShouldBeExactlyOneLayer case ShouldBeExactlyOneLayer
case ShouldBeAtLeastOneLayer case ShouldBeAtLeastOneLayer
case FailedToCreateDiskFile case FailedToCreateVmFile
} }
extension VMDirectory { extension VMDirectory {
@@ -31,8 +31,14 @@ extension VMDirectory {
if configLayers.count != 1 { if configLayers.count != 1 {
throw OCIError.ShouldBeExactlyOneLayer throw OCIError.ShouldBeExactlyOneLayer
} }
let configData = try await registry.pullBlob(configLayers.first!.digest) if !FileManager.default.createFile(atPath: configURL.path, contents: nil) {
try VMConfig(fromData: configData).save(toURL: configURL) throw OCIError.FailedToCreateVmFile
}
let configFile = try FileHandle(forWritingTo: configURL)
try await registry.pullBlob(configLayers.first!.digest) { buffer in
configFile.write(Data(buffer: buffer))
}
try configFile.close()
// Pull VM's disk layers and decompress them sequentially into a disk file // Pull VM's disk layers and decompress them sequentially into a disk file
let diskLayers = manifest.layers.filter { let diskLayers = manifest.layers.filter {
@@ -42,7 +48,7 @@ extension VMDirectory {
throw OCIError.ShouldBeAtLeastOneLayer throw OCIError.ShouldBeAtLeastOneLayer
} }
if !FileManager.default.createFile(atPath: diskURL.path, contents: nil) { if !FileManager.default.createFile(atPath: diskURL.path, contents: nil) {
throw OCIError.FailedToCreateDiskFile throw OCIError.FailedToCreateVmFile
} }
let disk = try FileHandle(forWritingTo: diskURL) let disk = try FileHandle(forWritingTo: diskURL)
let filter = try OutputFilter(.decompress, using: .lz4, bufferCapacity: Self.bufferSizeBytes) { data in let filter = try OutputFilter(.decompress, using: .lz4, bufferCapacity: Self.bufferSizeBytes) { data in
@@ -52,18 +58,23 @@ extension VMDirectory {
} }
// Progress // Progress
let diskCompressedSize: Int64 = Int64(diskLayers.map {$0.size}.reduce(0) {$0 + $1}) let diskCompressedSize: Int64 = Int64(diskLayers.map {
$0.size
}
.reduce(0) {
$0 + $1
})
let prettyDiskSize = String(format: "%.1f", Double(diskCompressedSize) / 1_000_000_000.0) let prettyDiskSize = String(format: "%.1f", Double(diskCompressedSize) / 1_000_000_000.0)
defaultLogger.appendNewLine("pulling disk (\(prettyDiskSize) GB compressed)...") defaultLogger.appendNewLine("pulling disk (\(prettyDiskSize) GB compressed)...")
let progress = Progress(totalUnitCount: diskCompressedSize) let progress = Progress(totalUnitCount: diskCompressedSize)
ProgressObserver(progress).log(defaultLogger) ProgressObserver(progress).log(defaultLogger)
for diskLayer in diskLayers { for diskLayer in diskLayers {
let diskData = try await registry.pullBlob(diskLayer.digest) try await registry.pullBlob(diskLayer.digest) { buffer in
try filter.write(diskData) let data = Data(buffer: buffer)
try filter.write(data)
// Progress progress.completedUnitCount += Int64(data.count)
progress.completedUnitCount += Int64(diskLayer.size) }
} }
try filter.finalize() try filter.finalize()
try disk.close() try disk.close()
@@ -77,11 +88,17 @@ extension VMDirectory {
if nvramLayers.count != 1 { if nvramLayers.count != 1 {
throw OCIError.ShouldBeExactlyOneLayer throw OCIError.ShouldBeExactlyOneLayer
} }
let nvramData = try await registry.pullBlob(nvramLayers.first!.digest) if !FileManager.default.createFile(atPath: nvramURL.path, contents: nil) {
try nvramData.write(to: nvramURL) throw OCIError.FailedToCreateVmFile
}
let nvram = try FileHandle(forWritingTo: nvramURL)
try await registry.pullBlob(nvramLayers.first!.digest) { buffer in
nvram.write(Data(buffer: buffer))
}
try nvram.close()
} }
func pushToRegistry(registry: Registry, references: [String]) async throws { func pushToRegistry(registry: Registry, references: [String]) async throws -> RemoteName {
var layers = Array<OCIManifestLayer>() var layers = Array<OCIManifestLayer>()
// Read VM's config and push it as blob // Read VM's config and push it as blob
@@ -92,7 +109,7 @@ extension VMDirectory {
// Progress // Progress
let diskSize = try FileManager.default.attributesOfItem(atPath: diskURL.path)[.size] as! Int64 let diskSize = try FileManager.default.attributesOfItem(atPath: diskURL.path)[.size] as! Int64
defaultLogger.appendNewLine("pushing disk... this will take a while...") defaultLogger.appendNewLine("pushing disk... this will take a while...")
let progress = Progress(totalUnitCount: diskSize) let progress = Progress(totalUnitCount: diskSize)
ProgressObserver(progress).log(defaultLogger) ProgressObserver(progress).log(defaultLogger)
@@ -102,7 +119,7 @@ extension VMDirectory {
let disk = try FileHandle(forReadingFrom: diskURL) let disk = try FileHandle(forReadingFrom: diskURL)
let compressingFilter = try InputFilter<Data>(.compress, using: .lz4, bufferCapacity: Self.bufferSizeBytes) { _ in let compressingFilter = try InputFilter<Data>(.compress, using: .lz4, bufferCapacity: Self.bufferSizeBytes) { _ in
let data = try disk.read(upToCount: Self.bufferSizeBytes) let data = try disk.read(upToCount: Self.bufferSizeBytes)
progress.completedUnitCount += Int64(data?.count ?? 0) progress.completedUnitCount += Int64(data?.count ?? 0)
return data return data
@@ -120,21 +137,27 @@ extension VMDirectory {
layers.append(OCIManifestLayer(mediaType: Self.nvramMediaType, size: nvram.count, digest: nvramDigest)) layers.append(OCIManifestLayer(mediaType: Self.nvramMediaType, size: nvram.count, digest: nvramDigest))
// Craft a stub OCI config for Docker Hub compatibility // Craft a stub OCI config for Docker Hub compatibility
struct OCIConfig: Encodable, Decodable { struct OCIConfig: Codable {
var architecture: String = "arm64" var architecture: String = "arm64"
var os: String = "darwin" var os: String = "darwin"
} }
let ociConfigJSON = try JSONEncoder().encode(OCIConfig()) let ociConfigJSON = try JSONEncoder().encode(OCIConfig())
let ociConfigDigest = try await registry.pushBlob(fromData: ociConfigJSON) let ociConfigDigest = try await registry.pushBlob(fromData: ociConfigJSON)
let ociConfigDescriptor = Descriptor(size: ociConfigJSON.count, digest: ociConfigDigest) let manifest = OCIManifest(
config: OCIManifestConfig(size: ociConfigJSON.count, digest: ociConfigDigest),
layers: layers
)
// Manifest // Manifest
for reference in references { for reference in references {
defaultLogger.appendNewLine("pushing manifest for \(reference)...") defaultLogger.appendNewLine("pushing manifest for \(reference)...")
_ = try await registry.pushManifest(reference: reference, config: ociConfigDescriptor, layers: layers) _ = try await registry.pushManifest(reference: reference, manifest: manifest)
} }
let pushedReference = Reference(digest: try manifest.digest())
return RemoteName(host: registry.baseURL.host!, namespace: registry.namespace, reference: pushedReference)
} }
} }
+7
View File
@@ -24,6 +24,13 @@ struct VMDirectory {
baseURL.lastPathComponent baseURL.lastPathComponent
} }
static func temporary() throws -> VMDirectory {
let tmpDir = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: false)
return VMDirectory(baseURL: tmpDir)
}
var initialized: Bool { var initialized: Bool {
FileManager.default.fileExists(atPath: configURL.path) && FileManager.default.fileExists(atPath: configURL.path) &&
FileManager.default.fileExists(atPath: diskURL.path) && FileManager.default.fileExists(atPath: diskURL.path) &&
+10 -1
View File
@@ -27,6 +27,11 @@ class VMStorageLocal {
return vmDir return vmDir
} }
func move(_ name: String, from: VMDirectory) throws {
_ = try FileManager.default.createDirectory(at: baseURL, withIntermediateDirectories: true)
_ = try FileManager.default.replaceItemAt(vmURL(name), withItemAt: from.baseURL)
}
func delete(_ name: String) throws { func delete(_ name: String) throws {
try FileManager.default.removeItem(at: vmURL(name)) try FileManager.default.removeItem(at: vmURL(name))
} }
@@ -36,9 +41,13 @@ class VMStorageLocal {
return try FileManager.default.contentsOfDirectory( return try FileManager.default.contentsOfDirectory(
at: baseURL, at: baseURL,
includingPropertiesForKeys: [.isDirectoryKey], includingPropertiesForKeys: [.isDirectoryKey],
options: .skipsSubdirectoryDescendants).map { url in options: .skipsSubdirectoryDescendants).compactMap { url in
let vmDir = VMDirectory(baseURL: url) let vmDir = VMDirectory(baseURL: url)
if !vmDir.initialized {
return nil
}
return (vmDir.name, vmDir) return (vmDir.name, vmDir)
} }
} catch { } catch {
+34 -15
View File
@@ -27,6 +27,17 @@ class VMStorageOCI {
return vmDir return vmDir
} }
func move(_ name: RemoteName, from: VMDirectory) throws{
let targetURL = vmURL(name)
// Pre-create intermediate directories (e.g. creates ~/.tart/cache/OCIs/github.com/org/repo/
// for github.com/org/repo:latest)
try FileManager.default.createDirectory(at: targetURL.deletingLastPathComponent(),
withIntermediateDirectories: true)
_ = try FileManager.default.replaceItemAt(targetURL, withItemAt: from.baseURL)
}
func delete(_ name: RemoteName) throws { func delete(_ name: RemoteName) throws {
try FileManager.default.removeItem(at: vmURL(name)) try FileManager.default.removeItem(at: vmURL(name))
} }
@@ -64,35 +75,43 @@ class VMStorageOCI {
func pull(_ name: RemoteName, registry: Registry) async throws { func pull(_ name: RemoteName, registry: Registry) async throws {
defaultLogger.appendNewLine("pulling manifest...") defaultLogger.appendNewLine("pulling manifest...")
let (manifest, manifestData) = try await registry.pullManifest(reference: name.reference) let (manifest, _) = try await registry.pullManifest(reference: name.reference.value)
var digestName = RemoteName(host: name.host, namespace: name.namespace,
reference: Reference(digest: try manifest.digest()))
// Create directory for manifest's digest
var digestName = name
digestName.reference = Digest.hash(manifestData)
if !exists(digestName) { if !exists(digestName) {
let vmDir = try create(digestName) let tmpVMDir = try VMDirectory.temporary()
try await vmDir.pullFromRegistry(registry: registry, manifest: manifest) try await withTaskCancellationHandler(operation: {
try await tmpVMDir.pullFromRegistry(registry: registry, manifest: manifest)
try move(digestName, from: tmpVMDir)
}, onCancel: {
try? FileManager.default.removeItem(at: tmpVMDir.baseURL)
})
} else { } else {
defaultLogger.appendNewLine("\(digestName.reference) image is already cached! creating a symlink...") defaultLogger.appendNewLine("\(digestName) image is already cached! creating a symlink...")
} }
// Create directory for reference if it's different if name != digestName {
if digestName != name {
// Overwrite the old symbolic link // Overwrite the old symbolic link
if FileManager.default.fileExists(atPath: vmURL(name).path) { try link(from: digestName, to: name)
try FileManager.default.removeItem(at: vmURL(name))
}
try FileManager.default.createSymbolicLink(at: vmURL(name), withDestinationURL: vmURL(digestName))
} }
} }
func link(from: RemoteName, to: RemoteName) throws {
if FileManager.default.fileExists(atPath: vmURL(to).path) {
try FileManager.default.removeItem(at: vmURL(to))
}
try FileManager.default.createSymbolicLink(at: vmURL(to), withDestinationURL: vmURL(from))
}
} }
extension URL { extension URL {
func appendingRemoteName(_ name: RemoteName) -> URL { func appendingRemoteName(_ name: RemoteName) -> URL {
var result: URL = self var result: URL = self
for pathComponent in (name.host + "/" + name.namespace + "/" + name.reference).split(separator: "/") { for pathComponent in (name.host + "/" + name.namespace + "/" + name.reference.value).split(separator: "/") {
result = result.appendingPathComponent(String(pathComponent)) result = result.appendingPathComponent(String(pathComponent))
} }
+93
View File
@@ -0,0 +1,93 @@
import XCTest
@testable import tart
final class RegistryTests: XCTestCase {
var registryRunner: RegistryRunner?
override func setUp() async throws {
try await super.setUp()
do {
registryRunner = try await RegistryRunner()
} catch {
try XCTSkipIf(ProcessInfo.processInfo.environment["CI"] == nil)
}
}
override func tearDown() async throws {
try await super.tearDown()
registryRunner = nil
}
var registry: Registry {
registryRunner!.registry
}
func testPushPullBlobSmall() async throws {
// Generate a simple blob
let pushedBlob = Data("The quick brown fox jumps over the lazy dog".utf8)
// Push it
let pushedBlobDigest = try await registry.pushBlob(fromData: pushedBlob)
XCTAssertEqual("sha256:d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592", pushedBlobDigest)
// Pull it
var pulledBlob = Data()
try await registry.pullBlob(pushedBlobDigest) { buffer in
pulledBlob.append(Data(buffer: buffer))
}
// Ensure that both blobs are identical
XCTAssertEqual(pushedBlob, pulledBlob)
}
func testPushPullBlobHuge() async throws {
// Generate a large enough blob
let fh = FileHandle(forReadingAtPath: "/dev/urandom")!
let largeBlobToPush = try fh.read(upToCount: 768 * 1024 * 1024)!
// Push it
let largeBlobDigest = try await registry.pushBlob(fromData: largeBlobToPush)
// Pull it
var pulledLargeBlob = Data()
try await registry.pullBlob(largeBlobDigest) { buffer in
pulledLargeBlob.append(Data(buffer: buffer))
}
// Ensure that both blobs are identical
XCTAssertEqual(largeBlobToPush, pulledLargeBlob)
}
func testPushPullManifest() async throws {
// Craft a basic config
struct OCIConfig: Codable {
var architecture: String = "arm64"
var os: String = "darwin"
}
let configData = try JSONEncoder().encode(OCIConfig())
let configDigest = try await registry.pushBlob(fromData: configData)
// Craft a basic layer
let layerData = Data("doesn't matter".utf8)
let layerDigest = try await registry.pushBlob(fromData: layerData)
// Craft a basic manifest and push it
let manifest = OCIManifest(
config: OCIManifestConfig(size: configData.count, digest: configDigest),
layers: [
OCIManifestLayer(mediaType: "application/octet-stream", size: layerData.count, digest: layerDigest)
]
)
let pushedManifestDigest = try await registry.pushManifest(reference: "latest", manifest: manifest)
// Ensure that the manifest pulled by tag matches with the one pushed above
let (pulledByTagManifest, _) = try await registry.pullManifest(reference: "latest")
XCTAssertEqual(manifest, pulledByTagManifest)
// Ensure that the manifest pulled by digest matches with the one pushed above
let (pulledByDigestManifest, _) = try await registry.pullManifest(reference: "\(pushedManifestDigest)")
XCTAssertEqual(manifest, pulledByDigestManifest)
}
}
+8 -2
View File
@@ -3,16 +3,22 @@ import XCTest
final class RemoteNameTests: XCTestCase { final class RemoteNameTests: XCTestCase {
func testTag() throws { func testTag() throws {
let expectedRemoteName = RemoteName(host: "ghcr.io", namespace: "a/b", reference: "latest") let expectedRemoteName = RemoteName(host: "ghcr.io", namespace: "a/b", reference: Reference(tag: "latest"))
XCTAssertEqual(expectedRemoteName, try RemoteName("ghcr.io/a/b:latest")) XCTAssertEqual(expectedRemoteName, try RemoteName("ghcr.io/a/b:latest"))
} }
func testComplexTag() throws {
let expectedRemoteName = RemoteName(host: "ghcr.io", namespace: "a/b", reference: Reference(tag: "1.2.3-RC-1"))
XCTAssertEqual(expectedRemoteName, try RemoteName("ghcr.io/a/b:1.2.3-RC-1"))
}
func testDigest() throws { func testDigest() throws {
let expectedRemoteName = RemoteName( let expectedRemoteName = RemoteName(
host: "ghcr.io", host: "ghcr.io",
namespace: "a/b", namespace: "a/b",
reference: "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" reference: Reference(digest: "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
) )
XCTAssertEqual(expectedRemoteName, XCTAssertEqual(expectedRemoteName,
+38
View File
@@ -0,0 +1,38 @@
import XCTest
@testable import tart
final class TokenResponseTests: XCTestCase {
func testBasic() throws {
let tokenResponseRaw = Data("{\"token\":\"some token\"}".utf8)
let tokenResponse = try TokenResponse.parse(fromData: tokenResponseRaw)
XCTAssertEqual(tokenResponse.token, "some token")
let expectedTokenExpiresAtRange = Date()...Date().addingTimeInterval(60)
XCTAssertTrue(expectedTokenExpiresAtRange.contains(tokenResponse.tokenExpiresAt))
XCTAssertTrue(tokenResponse.isValid)
}
func testExpirationBasic() throws {
let tokenResponseRaw = Data("{\"token\":\"some token\",\"expires_in\":2}".utf8)
let tokenResponse = try TokenResponse.parse(fromData: tokenResponseRaw)
XCTAssertEqual(tokenResponse.expiresIn, 2)
let expectedTokenExpiresAtRange = Date()...Date().addingTimeInterval(2)
XCTAssertTrue(expectedTokenExpiresAtRange.contains(tokenResponse.tokenExpiresAt))
XCTAssertTrue(tokenResponse.isValid)
_ = XCTWaiter.wait(for: [expectation(description: "Wait 3 seconds for the token to become invalid")], timeout: 2)
XCTAssertFalse(tokenResponse.isValid)
}
func testExpirationWithIssuedAt() throws {
let tokenResponseRaw = Data("{\"token\":\"some token\",\"expires_in\":3600,\"issued_at\":\"1970-01-01T00:00:00Z\"}".utf8)
let tokenResponse = try TokenResponse.parse(fromData: tokenResponseRaw)
XCTAssertEqual(Date(timeIntervalSince1970: 3600), tokenResponse.tokenExpiresAt)
XCTAssertFalse(tokenResponse.isValid)
}
}
+54
View File
@@ -0,0 +1,54 @@
import Foundation
@testable import tart
enum RegistryRunnerError: Error {
case DockerFailed(exitCode: Int32)
}
class RegistryRunner {
let containerID: String
let registry: Registry
static func dockerCmd(_ arguments: String...) throws -> String {
let stdoutPipe = Pipe()
let proc = Process()
proc.executableURL = URL(fileURLWithPath: "/usr/local/bin/docker")
proc.arguments = arguments
proc.standardOutput = stdoutPipe
try proc.run()
let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile()
proc.waitUntilExit()
if proc.terminationStatus != 0 {
throw RegistryRunnerError.DockerFailed(exitCode: proc.terminationStatus)
}
return String(data: stdoutData, encoding: .utf8) ?? ""
}
init() async throws {
// Start container
let container = try Self.dockerCmd("run", "-d", "--rm", "-p", "5000", "registry:2")
.trimmingCharacters(in: CharacterSet.newlines)
containerID = container
// Get forwarded port
let port = try Self.dockerCmd("inspect", containerID, "--format", "{{(index (index .NetworkSettings.Ports \"5000/tcp\") 0).HostPort}}")
.trimmingCharacters(in: CharacterSet.newlines)
registry = try Registry(urlComponents: URLComponents(string: "http://127.0.0.1:\(port)/v2/")!,
namespace: "vm-image")
// Wait for the Docker Registry to start
while ((try? await registry.ping()) == nil) {
try await Task.sleep(nanoseconds: 100_000_000)
}
}
deinit {
_ = try! Self.dockerCmd("kill", containerID)
}
}