-
Notifications
You must be signed in to change notification settings - Fork 471
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Write GitHub Actions workflow files in Kotlin instead of YAML #1630
Open
Vampire
wants to merge
30
commits into
spockframework:master
Choose a base branch
from
Vampire:kotlin-workflow-files
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
b6e5d76
Write GitHub Actions workflow files in Kotlin instead of Yaml
Vampire d22743b
Update to latest github-workflows-kt version and some improvements
Vampire b61b0eb
Move preprocessWorkflow task creation to build-logic
Vampire bf8ef4e
Verify consistency of all workflow YAMLs on pull requests and branches
Vampire bf79409
yaml => yml
Vampire d4f1d72
Use programmatic logic for variants and java versions
Vampire 690d1a5
Use one central location to define the java versions and variants
Vampire 0247fe2
Improve all script consistency check
Vampire 955f8cb
Update github-workflows-kt to v0.44.0
Vampire 8606f7b
Review feedback
Vampire 170892d
Review feedback v2
Vampire fc23128
Catch up changes from master
Vampire 7049f8d
Update github-workflows-kt version to 0.50.0
Vampire 93aa39e
Set group for GHA tasks to 'github actions'
Vampire f2e5f71
Catch up changes from master
Vampire 6119c3f
Catch up changes from master
Vampire 04d9237
Update github-workflows-kt version to 1.3.0
Vampire c3c1406
Update github-workflows-kt version to 1.4.0
Vampire 23b171a
Catch up changes from master
Vampire 80aae4f
Update github-workflows-kt version to 1.12.0
Vampire 4e02013
Catch up changes from master
Vampire 17bff51
Update Kotlin code style settings
Vampire 13a6041
Catch up changes from master
Vampire c2c8131
Catch up changes from master
Vampire d8ea1ce
Catch up changes from master
Vampire 267d269
Only use zulu on arm macOS
Vampire c02372f
Update github-workflows-kt version to 2.0.0
Vampire 1520e8d
Update github-workflows-kt version to 2.3.0 and Maven-based bindings
Vampire 2ffa401
Fix typo in workflow readme
Vampire 9ad73eb
Improve all-workflows consistency check
Vampire File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
== The YAML workflow files vs. the `*.main.kts` files | ||
|
||
The YAML workflow files are generated from the `*.main.kts` files. | ||
|
||
These use the https://github.com/typesafegithub/github-workflows-kt[github-workflows-kt] | ||
Kotlin DSL library to conveniently and type-safely write GitHub Action workflow files. | ||
|
||
As there is no official built-in support in GitHub Actions yet until | ||
https://github.com/orgs/community/discussions/15904 is considered, the YAML files | ||
need to be generated manually. | ||
|
||
There is a safeguard check in all the generated files that this is not forgotten. | ||
Running a workflow where the according `*.main.kts` produces a different output will | ||
fail the execution. Additionally, the workflow that runs for pull requests checks | ||
the consistency of all the YAML files as not all are run for pull requests. | ||
|
||
|
||
|
||
== Ways to generate the YAML workflow files | ||
|
||
There are multiple ways to generate the YAML files and all of them are fine, | ||
but be aware of the last one of the caveats below if you are not using the Gradle method: | ||
|
||
* If you are in a `sh` derivate like e.g. `bash` and Kotlin is installed and | ||
available in the `PATH`, you can just call the `*.main.kts` script like any | ||
other shell script: | ||
+ | ||
[source,bash] | ||
---- | ||
$ ./release.main.kts | ||
---- | ||
|
||
* If Kotlin is installed somewhere you can call it with the `*.main.kts` script | ||
as argument: | ||
+ | ||
[source,bash] | ||
---- | ||
$ path/to/kotlin release.main.kts | ||
---- | ||
|
||
* From the IDE you can create a run configuration that executes the `*.main.kts` script. | ||
|
||
* There is a Gradle task `preprocessWorkflows` that generates all YAML files from the | ||
according `*.main.kts` files. Additionally, there is also one task per workflow to | ||
only generate that one: | ||
+ | ||
[source,bash] | ||
---- | ||
$ ./gradlew preprocessReleaseWorkflow | ||
$ ./gradlew preprocessWorkflows | ||
---- | ||
|
||
|
||
|
||
== Caveats | ||
|
||
There are currently three known caveats with the approach we follow. | ||
|
||
* https://youtrack.jetbrains.com/issue/KTIJ-16532 | ||
+ | ||
If you navigate to a file in the dependencies, only a decompiled file is opened, | ||
even though the source JAR would be available. Also the quick documentation is missing. | ||
+ | ||
This can easily by mitigated by attaching the library to the normal project | ||
dependencies while having the need to navigate the source files or while editing them, | ||
which makes them properly viewable and documentation displayable in the editor. | ||
|
||
* https://youtrack.jetbrains.com/issue/KTIJ-14580 | ||
+ | ||
We use `@file:Import` to reduce code duplication by having common code in a common file. | ||
Unfortunately, this triggers a Kotlin IntelliJ plugin bug where the imported file cannot | ||
be loaded properly and so the things supplied by it like dependencies or common functions | ||
are not available. This makes most of the workflow `*.main.kts` files red as hell in the | ||
IDE currently. | ||
+ | ||
To reduce risk for eye-cancer while reading the `*.main.kts` scripts or to be able to | ||
sanely edit them, temporarily add the `@file:DependsOn` from the imported file to the | ||
importing file and wait a second, then remove the line again once you are done. | ||
|
||
* https://youtrack.jetbrains.com/issue/KT-42101 | ||
+ | ||
We use `@file:Import` to reduce code duplication by having common code in a common file. | ||
Unfortunately, this triggers a Kotlin bug where the compilation cache becomes confused | ||
if the imported file is changed without the importing file being changed too. | ||
+ | ||
If only the imported file is changed, it could happen that an old version is used, | ||
or it could also happen that classes added by a `@file:DependsOn` in the imported file | ||
are not available to the importing file. So if there was a change in the imported file, | ||
you either need to also change the importing file, or to properly execute the script, | ||
you need to delete the stale entry from the compilation cache which can be found at for example | ||
`~/.cache/main.kts.compiled.cache/` on Linux and `%LOCALAPPDATA%\main.kts.compiled.cache\` | ||
on Windows. Alternatively, you can also delete the whole cache directory. | ||
+ | ||
Another option is to disable the compilation cache for the execution by setting the | ||
environment variable `KOTLIN_MAIN_KTS_COMPILED_SCRIPTS_CACHE_DIR` or the system property | ||
`kotlin.main.kts.compiled.scripts.cache.dir` to an empty value, depending on the run | ||
method you chose. The Gradle tasks already do that, so when using the Gradle tasks you | ||
do not have this problem and it just works. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
#!/usr/bin/env kotlin | ||
|
||
/* | ||
* Copyright 2023 the original author or authors. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* https://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
@file:Import("common.main.kts") | ||
@file:Repository("https://bindings.krzeminski.it/") | ||
@file:DependsOn("actions:checkout:v4") | ||
@file:DependsOn("codecov:codecov-action:v4") | ||
|
||
import io.github.typesafegithub.workflows.actions.actions.Checkout | ||
import io.github.typesafegithub.workflows.actions.actions.Checkout.FetchDepth | ||
import io.github.typesafegithub.workflows.actions.codecov.CodecovAction | ||
import io.github.typesafegithub.workflows.domain.Concurrency | ||
import io.github.typesafegithub.workflows.domain.RunnerType | ||
import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest | ||
import io.github.typesafegithub.workflows.domain.triggers.MergeGroup | ||
import io.github.typesafegithub.workflows.domain.triggers.PullRequest | ||
import io.github.typesafegithub.workflows.domain.triggers.Push | ||
import io.github.typesafegithub.workflows.dsl.expressions.Contexts.github | ||
import io.github.typesafegithub.workflows.dsl.expressions.expr | ||
import io.github.typesafegithub.workflows.dsl.workflow | ||
|
||
workflow( | ||
name = "Verify Branches and PRs", | ||
on = listOf( | ||
Push( | ||
branchesIgnore = listOf( | ||
"master", | ||
"gh-pages" | ||
) | ||
), | ||
PullRequest(), | ||
MergeGroup() | ||
), | ||
sourceFile = __FILE__, | ||
targetFileName = "${__FILE__.name.substringBeforeLast(".main.kts")}.yml", | ||
// https://stackoverflow.com/a/72408109/16358266 | ||
concurrency = Concurrency( | ||
group = "${expr { github.workflow }}-${expr("${github.eventPullRequest.pull_request.number} || ${github.ref}")}", | ||
cancelInProgress = true | ||
) | ||
) { | ||
job( | ||
id = "check_all_workflow_yaml_consistency", | ||
name = "Check all Workflow YAML Consistency", | ||
runsOn = UbuntuLatest | ||
) { | ||
uses( | ||
name = "Checkout Repository", | ||
action = Checkout() | ||
) | ||
run( | ||
name = "Regenerate all Workflow YAMLs", | ||
command = """find .github/workflows -mindepth 1 -maxdepth 1 -name '*.main.kts' -exec {} \;""" | ||
) | ||
run( | ||
name = "Check for Modifications", | ||
command = """ | ||
git add --intent-to-add . | ||
git diff --exit-code | ||
""".trimIndent() | ||
) | ||
} | ||
|
||
job( | ||
id = "build-and-verify", | ||
name = "Build and Verify", | ||
runsOn = RunnerType.Custom(expr(Matrix.operatingSystem)), | ||
strategy = Strategy( | ||
matrix = Matrix.full | ||
) | ||
) { | ||
uses( | ||
name = "Checkout Repository", | ||
action = Checkout( | ||
// Codecov needs fetch-depth > 1 | ||
fetchDepth = FetchDepth.Value(2) | ||
) | ||
) | ||
uses( | ||
name = "Set up JDKs", | ||
action = SetupBuildEnv( | ||
additionalJavaVersion = expr(Matrix.javaVersion) | ||
) | ||
) | ||
run( | ||
name = "Build Spock", | ||
command = listOf( | ||
"./gradlew", | ||
"--stacktrace", | ||
"ghActionsBuild", | ||
""""-Dvariant=${expr(Matrix.variant)}"""", | ||
""""-DjavaVersion=${expr(Matrix.javaVersion)}"""" | ||
).joinToString(" "), | ||
// secrets are not injected for pull requests | ||
env = commonCredentials | ||
) | ||
uses( | ||
name = "Upload to Codecov.io", | ||
action = CodecovAction() | ||
) | ||
} | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we keep the
.yml
extension for the generated files, so that we can actually see the diff?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you can manually configure the target file name.
I changed it to
...yml
, but maybe we want to return to the default after you verified the changes to not have the unnecessary configuration?