Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamline the swiftly init process #177

Merged
merged 51 commits into from
Dec 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
e4efbeb
DRAFT: Swiftly proxies
cmcgee1024 Aug 10, 2024
3b5c404
Add a swiftly install workflow where the version comes from the .swif…
cmcgee1024 Aug 10, 2024
956256f
update design to make auto-installation an error instead
cmcgee1024 Aug 20, 2024
375df29
provide a mechanism to find the currently in-use toolchain physical l…
cmcgee1024 Aug 20, 2024
ed68a9e
add more details about the selector prefix, and methods to escape
cmcgee1024 Aug 21, 2024
c745f4c
Restructure the PR to move the selector syntax from the proxies to a …
cmcgee1024 Aug 29, 2024
560236d
Implement proxy mechanism with dynamic toolchain selection
cmcgee1024 Sep 4, 2024
ae29e88
Merge branch 'main' of github.com:cmcgee1024/swiftly into proxies-design
cmcgee1024 Sep 5, 2024
ce0d27a
Rewrite the select toolchain function with a type for the selection r…
cmcgee1024 Sep 5, 2024
66dd459
Update the documentation
cmcgee1024 Sep 5, 2024
2df7357
Create a swiftly run command
cmcgee1024 Sep 6, 2024
8976bba
Fix empty command case with a single ++++
cmcgee1024 Sep 7, 2024
3caab66
Write run command and proxy tests
cmcgee1024 Sep 9, 2024
31f4327
Regenerate the cli reference documentation
cmcgee1024 Sep 9, 2024
16caf80
Fix design document discrepancies and add install proxy argument tests
cmcgee1024 Sep 9, 2024
b2aa165
Update the list command to decorate default, and in-use toolchains
cmcgee1024 Sep 10, 2024
7504dd3
Update list tests to check for in use and default labels
cmcgee1024 Sep 10, 2024
0e7b661
Make the version argument optional in the install subcommand
cmcgee1024 Sep 11, 2024
43d620a
Fix case of empty bin directory when checking for overwrite
cmcgee1024 Sep 12, 2024
2e3c59d
Remove +install selector option from swift run in favour of regular `…
cmcgee1024 Sep 20, 2024
9dd640c
Import GPG keys on every install to get new signing keys from swift.org
cmcgee1024 Sep 17, 2024
5e615ea
Make recommended documentation changes.
cmcgee1024 Oct 18, 2024
10a0856
Merge branch 'main' of github.com:cmcgee1024/swiftly into proxies-design
cmcgee1024 Oct 18, 2024
6d6050e
Provide a better error message on swiftly install with no version
cmcgee1024 Oct 19, 2024
883e638
Add new Linux platforms: Ubuntu 2404,2310, Debian 12, Fedora 39
cmcgee1024 Oct 19, 2024
0cc129c
Add the new entries to the --platform option
cmcgee1024 Oct 19, 2024
afbd797
Pull list of linux platforms out into a var.
cmcgee1024 Oct 19, 2024
d301ef3
Remove all of the platform def environment variables to force auto-de…
cmcgee1024 Oct 19, 2024
393f9d5
Remove last remnants of the special platform environment variables fr…
cmcgee1024 Oct 20, 2024
6989796
Update README, and add documentation for the new run subcommand
cmcgee1024 Oct 20, 2024
6fc24f0
Remove mention of CentOS from the README.md since it is not particula…
cmcgee1024 Oct 20, 2024
71df3a6
Streamline the swiftly init process
cmcgee1024 Oct 20, 2024
4d833f8
Add more detail to the getting started guide
cmcgee1024 Oct 23, 2024
bc83ee5
Remove the word system from the missing dependencies message
cmcgee1024 Oct 23, 2024
be65db1
Add system package dependency list for Ubuntu 23.10
cmcgee1024 Oct 23, 2024
646b5cb
Rework the getting started guide to have a platform selector for linu…
cmcgee1024 Oct 23, 2024
4e7ca7c
Merge branch 'main' of github.com:cmcgee1024/swiftly into add_new_lin…
cmcgee1024 Oct 25, 2024
a08faad
Permit bare swiftly command to start the init workflow when not insta…
cmcgee1024 Oct 29, 2024
65da6d8
Merge branch 'add_new_linux_platforms' of github.com:cmcgee1024/swift…
cmcgee1024 Oct 29, 2024
970943f
Fix array index problem
cmcgee1024 Oct 29, 2024
c43fba2
Use result to simplify proxy init check
cmcgee1024 Oct 29, 2024
bce8ab6
Quiet the macOS installer messages behind the verbose flag
cmcgee1024 Oct 30, 2024
bbe7277
Merge branch 'main' of github.com:cmcgee1024/swiftly into streamlined…
cmcgee1024 Nov 21, 2024
35a05d0
Fix merge remnants and reformat
cmcgee1024 Nov 21, 2024
d732a84
Fixes for the system managed detection and installation logic
cmcgee1024 Nov 21, 2024
34e4021
Update Sources/SwiftlyCore/Platform.swift
cmcgee1024 Nov 22, 2024
55fc274
Refactor find/install swiftly binary into two functions
cmcgee1024 Nov 22, 2024
942c4e4
Re-introduce the bare swiftly invocation in an uninstalled state redi…
cmcgee1024 Nov 22, 2024
e7ab371
Propagate the verbose flag to the pkgutil call when installing in non…
cmcgee1024 Dec 5, 2024
73a77ea
Merge branch 'main' of github.com:cmcgee1024/swiftly into streamlined…
cmcgee1024 Dec 5, 2024
be095e5
Add proxy exclusion for asking swiftly for help or dumping help json
cmcgee1024 Dec 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions Documentation/SwiftlyDocs.docc/automated-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,26 @@ Swiftly can be installed automatically in places like build/CI systems.

This guide will help you to script to the installation of swiftly and toolchains so that it can be unattended. We assume that you have working understanding of your build system. The examples are based on a typical Unix environment.

First, download a swiftly binary from a trusted source, such as your artifact repository, or a well-known website for the operating system (e.g. Linux) and processor architecture (e.g. arm64, or x86_64). Here's an example using the popular curl command.
First, download the swiftly binary from swift.org for your operating system (e.g. Linux) and processor architecture (e.g. arm64, or x86_64). Here's an example using the popular curl command.

```
curl -L <trusted_location_of_swiftly> > swiftly
curl -L <location_of_swiftly_swift_org> > swiftly.tar.gz

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

location_of_swiftly_swift_org

Does this exist? 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will once these website update PR's are merged with a released version of swiftly:

swiftlang/swift-org-website#828
swiftlang/swift-org-website#830

tar zxf swiftly.tar.gz
```

On macOS you can download the pkg file and extract it like this from the command-line:

```
curl -L <location_of_swiftly_swift_org> > swiftly.pkg
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory
```

> Tip: If you are using Linux you will need the "ca-certificates" package for the root certificate authorities that will establish the trust that swiftly needs to make API requests that it needs. This package is frequently pre-installed on end-user environments, but may not be present in more minimal installations.

Once swiftly is downloaded you can run the init subcommand to finish the installation. This command will use the default initialization options and proceed without prompting.
Once swiftly is downloaded you can run the init subcommand to finish the installation. This command will print verbose outputs, assume yes for all prompts, and skip the automatic installation of the latest swift toolchain:

```
./swiftly init --assume-yes
./swiftly init --verbose --assume-yes --skip-install # the swiftly binary is extracted to ~/local/bin/swiftly on macOS
```

Swiftly is installed, but the current shell may not yet be updated with the new environment variables, such as the PATH. The init command prints instructions on how to update the current shell environment without opening a new shell. This is an example of the output taken from Linux, but the details might be different for other OSes, username, or shell.
Expand Down
49 changes: 32 additions & 17 deletions Documentation/SwiftlyDocs.docc/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,46 @@
# Getting Started with Swiftly

To download swiftly and install Swift, run the following in your terminal, then follow the on-screen instructions:
Start using swiftly and swift.

```
curl -L https://swiftlang.github.io/swiftly/swiftly-install.sh | bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason we can't offer the script as an alternative? Removing it completely makes install Swiftly and Swift a reasonably amount more complicated as it puts the onus on the user to work out what OS and architecture they're using, which they can get wrong and make it seem like Swift doesn't work.

Additionally this makes building tooling on top of Swiftly significantly more complicated and each tool has to do the same logic of working out what system it's running on and going to find the correct binary which is a lot of duplicated work. It would be good to have that centralised in the tool.

Finally, this reduces the 'copy and paste this one line' installation for Swift that we have had up to now, making the experience on installing Swift on different systems a step back IMO compared to say rustup.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are number of ways that we're hoping to address these as we can't use a curl | bash approach.

For the regular user install workflow, the website can help to find them the most likely swiftly download candidate for their OS and architecture. More details can be found in the PR here: swiftlang/swift-org-website#830

We can also add a very short guide to help them on the next steps. Feedback is welcome on that PR.

The download for macOS brings them to the pkg installer for the easy flow. It can also be invoked headlessly on the command-line either in a system directory or the user's home directory. Details are in the docs.

Since swiftly is a Linux binary that can work on a wide variety of Linux distributions, it takes on the responsibility of Linux flavour detection, and configuration. The user needs only to untar it and run swiftly wherever it is to continue.

For automated systems, there's a document here that goes over how one would automatically install swiftly. Feedback is welcome on this document as part of a separate PR/issue: https://swiftpackageindex.com/swiftlang/swiftly/main/documentation/swiftlydocs/automated-install

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the slightly longer term, there is also the possibility of providing swiftly in system package managers to make this even easier. Swiftly is able to be run from a system location.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are number of ways that we're hoping to address these as we can't use a curl | bash approach.
We can no longer use the install script for some reason? I agree with Tim that the one liner approach seems a bit more user friendly as it handles environment detection

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that in the near term the website can do much to ease the new user flow to compensate, such as making sure that they have the right executable, escape hatch to the full list of executables. The URL when it's finally decided for the website, can be made to be friendly to uname -m and uname -s. System packages of swiftly will be an accelerator as well, once they are available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But what's the actual reason "we can't use a curl | bash approach"? I fully support the plan to offer an executable that does more work and make it possible to install without requiring a shell script, but removing it all together as an option feels like a step backwards

```
To get started with swiftly you can download it from [swift.org](https://swift.org/download), and extract the package.

Alternatively, you can download the swiftly binary and install itself like this:
@TabNavigator {
@Tab("Linux") {
If you are using Linux then you can verify and extract the archive like this:

```
swiftly init
```
```
sha256sum swiftly-x.y.z.tar.gz # Check that the hash matches what's reported on swift.org
tar zxf swiftly-x.y.z.tar.gz
```

Once swiftly is installed you can use it to install the latest available swift toolchain like this:
Now run swiftly init to finish the installation:

```
$ swiftly install latest
```
./swiftly init
```
}

Fetching the latest stable Swift release...
Installing Swift 5.8.1
Downloaded 488.5 MiB of 488.5 MiB
Extracting toolchain...
Swift 5.8.1 installed successfully!
@Tab("macOS") {
On macOS you can either run the pkg installer from the command-line like this or just run the package by double-clicking on it (not recommended):

```
installer -pkg swift-x.y.z.pkg -target CurrentUserHomeDirectory
```

Now run swiftly init to finish the installation:

```
$HOME/usr/local/bin/swiftly init
```
}
}

Swiftly will install itself and download the latest available Swift toolchain. Follow the prompts for any additional steps. Once everything is done you can begin using swift.

```
$ swift --version

Swift version 5.8.1 (swift-5.8.1-RELEASE)
Swift version 6.0.1 (swift-6.0.1-RELEASE)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few lines down the doc references swift build with a commend saying latest (5.8.1) toolchain, probably just want to update that comment to 6.0.1 too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can raise another PR to fresh the version numbers in the documentation, or make them more version independent. Suggestions are welcome on how best to handle this problem of balancing accuracy and maintainability.

Target: x86_64-unknown-linux-gnu

$ swift build # Build with the latest (5.8.1) toolchain
Expand Down
54 changes: 47 additions & 7 deletions Documentation/SwiftlyDocs.docc/swiftly-cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ swiftly [--version] [--help]
Install a new toolchain.

```
swiftly install [<version>] [--use] [--verify|no-verify] [--post-install-file=<post-install-file>] [--assume-yes] [--version] [--help]
swiftly install [<version>] [--use] [--verify|no-verify] [--post-install-file=<post-install-file>] [--assume-yes] [--verbose] [--version] [--help]
```

**version:**
Expand Down Expand Up @@ -81,6 +81,11 @@ written to this file as commands that can be run after the installation.
*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**--version:**

*Show the version.*
Expand Down Expand Up @@ -143,7 +148,7 @@ Note that listing available snapshots before the latest release (major and minor
Set the in-use toolchain. If no toolchain is provided, print the currently in-use toolchain, if any.

```
swiftly use [--print-location] [--global-default] [--assume-yes] [<toolchain>] [--version] [--help]
swiftly use [--print-location] [--global-default] [--assume-yes] [--verbose] [<toolchain>] [--version] [--help]
```

**--print-location:**
Expand All @@ -161,6 +166,11 @@ swiftly use [--print-location] [--global-default] [--assume-yes] [<toolchain>] [
*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**toolchain:**

*The toolchain to use.*
Expand Down Expand Up @@ -210,7 +220,7 @@ Likewise, the latest snapshot associated with a given development branch can be
Remove an installed toolchain.

```
swiftly uninstall <toolchain> [--assume-yes] [--version] [--help]
swiftly uninstall <toolchain> [--assume-yes] [--verbose] [--version] [--help]
```

**toolchain:**
Expand Down Expand Up @@ -249,6 +259,11 @@ Finally, all installed toolchains can be uninstalled by specifying 'all':
*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**--version:**

*Show the version.*
Expand Down Expand Up @@ -309,7 +324,7 @@ The installed snapshots for a given devlopment branch can be listed by specifyin
Update an installed toolchain to a newer version.

```
swiftly update [<toolchain>] [--assume-yes] [--verify|no-verify] [--post-install-file=<post-install-file>] [--version] [--help]
swiftly update [<toolchain>] [--assume-yes] [--verbose] [--verify|no-verify] [--post-install-file=<post-install-file>] [--version] [--help]
```

**toolchain:**
Expand Down Expand Up @@ -355,6 +370,11 @@ A specific snapshot toolchain can be updated by including the date:
*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**--verify|no-verify:**

*Verify the toolchain's PGP signature before proceeding with installation.*
Expand Down Expand Up @@ -385,7 +405,7 @@ written to this file as commands that can be run after the installation.
Perform swiftly initialization into your user account.

```
swiftly init [--no-modify-profile] [--overwrite] [--platform=<platform>] [--assume-yes] [--version] [--help]
swiftly init [--no-modify-profile] [--overwrite] [--platform=<platform>] [--skip-install] [--assume-yes] [--verbose] [--version] [--help]
```

**--no-modify-profile:**
Expand All @@ -400,14 +420,24 @@ swiftly init [--no-modify-profile] [--overwrite] [--platform=<platform>] [--assu

**--platform=\<platform\>:**

*Specify the current Linux platform for swiftly.*
*Specify the current Linux platform for swiftly*


**--skip-install:**

*Skip installing the latest toolchain*


**--assume-yes:**

*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**--version:**

*Show the version.*
Expand All @@ -425,9 +455,19 @@ swiftly init [--no-modify-profile] [--overwrite] [--platform=<platform>] [--assu
Update the version of swiftly itself.

```
swiftly self-update [--version] [--help]
swiftly self-update [--assume-yes] [--verbose] [--version] [--help]
```

**--assume-yes:**

*Disable confirmation prompts by assuming 'yes'*


**--verbose:**

*Enable verbose reporting from swiftly*


**--version:**

*Show the version.*
Expand Down
6 changes: 1 addition & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,8 @@ Ongoing maintenance and stewardship of this project is led by the [SSWG](https:/

### Installation

To download swiftly and install Swift, run the following in your terminal, then follow the on-screen instructions.
```
curl -L https://swiftlang.github.io/swiftly/swiftly-install.sh | bash
```
Download the swiftly package from [swift.org](https://swift.org/download) and it can install itself with init:

Alternatively, you can download the swiftly binary and it can install itself:
```
swiftly init
```
Expand Down
15 changes: 11 additions & 4 deletions Sources/LinuxPlatform/Linux.swift
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,7 @@ public struct Linux: Platform {
}
}

public func install(from tmpFile: URL, version: ToolchainVersion) throws {
public func install(from tmpFile: URL, version: ToolchainVersion, verbose: Bool) throws {
guard tmpFile.fileExists() else {
throw Error(message: "\(tmpFile) doesn't exist")
}
Expand All @@ -348,7 +348,14 @@ public struct Linux: Platform {
let relativePath = name.drop { c in c != "/" }.dropFirst()

// prepend /path/to/swiftlyHomeDir/toolchains/<toolchain> to each file name
return toolchainDir.appendingPathComponent(String(relativePath))
let destination = toolchainDir.appendingPathComponent(String(relativePath))

if verbose {
SwiftlyCore.print("\(destination.path)")
}

// prepend /path/to/swiftlyHomeDir/toolchains/<toolchain> to each file name
return destination
}
}

Expand Down Expand Up @@ -390,7 +397,7 @@ public struct Linux: Platform {
FileManager.default.temporaryDirectory.appendingPathComponent("swiftly-\(UUID())")
}

public func verifySignature(httpClient: SwiftlyHTTPClient, archiveDownloadURL: URL, archive: URL) async throws {
public func verifySignature(httpClient: SwiftlyHTTPClient, archiveDownloadURL: URL, archive: URL, verbose: Bool) async throws {
SwiftlyCore.print("Downloading toolchain signature...")
let sigFile = self.getTempFilePath()
let _ = FileManager.default.createFile(atPath: sigFile.path, contents: nil)
Expand All @@ -405,7 +412,7 @@ public struct Linux: Platform {

SwiftlyCore.print("Verifying toolchain signature...")
do {
try self.runProgram("gpg", "--verify", sigFile.path, archive.path)
try self.runProgram("gpg", "--verify", sigFile.path, archive.path, quiet: !verbose)
} catch {
throw Error(message: "Signature verification failed: \(error).")
}
Expand Down
13 changes: 8 additions & 5 deletions Sources/MacOSPlatform/MacOS.swift
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ public struct MacOS: Platform {
nil
}

public func install(from tmpFile: URL, version: ToolchainVersion) throws {
public func install(from tmpFile: URL, version: ToolchainVersion, verbose: Bool) throws {
guard tmpFile.fileExists() else {
throw Error(message: "\(tmpFile) doesn't exist")
}
Expand All @@ -60,23 +60,26 @@ public struct MacOS: Platform {

if SwiftlyCore.mockedHomeDir == nil {
SwiftlyCore.print("Installing package in user home directory...")
try runProgram("installer", "-pkg", tmpFile.path, "-target", "CurrentUserHomeDirectory")
try runProgram("installer", "-verbose", "-pkg", tmpFile.path, "-target", "CurrentUserHomeDirectory", quiet: !verbose)
} else {
// In the case of a mock for testing purposes we won't use the installer, perferring a manual process because
// the installer will not install to an arbitrary path, only a volume or user home directory.
SwiftlyCore.print("Expanding pkg...")
let tmpDir = self.getTempFilePath()
let toolchainDir = self.swiftlyToolchainsDir.appendingPathComponent("\(version.identifier).xctoolchain", isDirectory: true)
if !toolchainDir.fileExists() {
try FileManager.default.createDirectory(at: toolchainDir, withIntermediateDirectories: false)
}
try runProgram("pkgutil", "--expand", tmpFile.path, tmpDir.path)
try runProgram("pkgutil", "--verbose", "--expand", tmpFile.path, tmpDir.path, quiet: !verbose)
// There's a slight difference in the location of the special Payload file between official swift packages
// and the ones that are mocked here in the test framework.
var payload = tmpDir.appendingPathComponent("Payload")
if !payload.fileExists() {
payload = tmpDir.appendingPathComponent("\(version.identifier)-osx-package.pkg/Payload")
}
try runProgram("tar", "-C", toolchainDir.path, "-xf", payload.path)

SwiftlyCore.print("Untarring pkg Payload...")
try runProgram("tar", "-C", toolchainDir.path, "-xvf", payload.path, quiet: !verbose)
}
}

Expand Down Expand Up @@ -146,7 +149,7 @@ public struct MacOS: Platform {
FileManager.default.temporaryDirectory.appendingPathComponent("swiftly-\(UUID()).pkg")
}

public func verifySignature(httpClient _: SwiftlyHTTPClient, archiveDownloadURL _: URL, archive _: URL) async throws {
public func verifySignature(httpClient _: SwiftlyHTTPClient, archiveDownloadURL _: URL, archive _: URL, verbose _: Bool) async throws {
// No signature verification is required on macOS since the pkg files have their own signing
// mechanism and the swift.org downloadables are trusted by stock macOS installations.
}
Expand Down
Loading
Loading