-
Notifications
You must be signed in to change notification settings - Fork 470
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Write GitHub Actions workflow files in Kotlin instead of YAML #1630
Merged
leonard84
merged 31 commits into
spockframework:master
from
Vampire:kotlin-workflow-files
Dec 27, 2024
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
e617f87
Write GitHub Actions workflow files in Kotlin instead of Yaml
Vampire 0ffedb5
Update to latest github-workflows-kt version and some improvements
Vampire c5a6475
Move preprocessWorkflow task creation to build-logic
Vampire 052d0de
Verify consistency of all workflow YAMLs on pull requests and branches
Vampire 0e3c4ca
yaml => yml
Vampire e857d0b
Use programmatic logic for variants and java versions
Vampire dc6f471
Use one central location to define the java versions and variants
Vampire fe53ff3
Improve all script consistency check
Vampire 613c93c
Update github-workflows-kt to v0.44.0
Vampire 82ce9a0
Review feedback
Vampire 75a4bfa
Review feedback v2
Vampire 1c5f29f
Catch up changes from master
Vampire 92fa4b3
Update github-workflows-kt version to 0.50.0
Vampire 3db6f1c
Set group for GHA tasks to 'github actions'
Vampire e734f06
Catch up changes from master
Vampire 9e9d977
Catch up changes from master
Vampire 7f6a40d
Update github-workflows-kt version to 1.3.0
Vampire 350f8b0
Update github-workflows-kt version to 1.4.0
Vampire c630008
Catch up changes from master
Vampire f8db6fd
Update github-workflows-kt version to 1.12.0
Vampire a68446b
Catch up changes from master
Vampire 34898a0
Update Kotlin code style settings
Vampire 5e0f01d
Catch up changes from master
Vampire b779792
Catch up changes from master
Vampire 76850f6
Catch up changes from master
Vampire 3c9466f
Only use zulu on arm macOS
Vampire 816c4c8
Update github-workflows-kt version to 2.0.0
Vampire 0400056
Update github-workflows-kt version to 2.3.0 and Maven-based bindings
Vampire 73dfa6c
Fix typo in workflow readme
Vampire 1b53632
Improve all-workflows consistency check
Vampire 5d5dfdd
Catch up changes from master
Vampire File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
== The YAML workflow files vs. the `*.main.kts` files | ||
|
||
The YAML workflow files are generated from the `*.main.kts` files. | ||
|
||
These use the https://github.com/typesafegithub/github-workflows-kt[github-workflows-kt] | ||
Kotlin DSL library to conveniently and type-safely write GitHub Action workflow files. | ||
|
||
As there is no official built-in support in GitHub Actions yet until | ||
https://github.com/orgs/community/discussions/15904 is considered, the YAML files | ||
need to be generated manually. | ||
|
||
There is a safeguard check in all the generated files that this is not forgotten. | ||
Running a workflow where the according `*.main.kts` produces a different output will | ||
fail the execution. Additionally, the workflow that runs for pull requests checks | ||
the consistency of all the YAML files as not all are run for pull requests. | ||
|
||
|
||
|
||
== Ways to generate the YAML workflow files | ||
|
||
There are multiple ways to generate the YAML files and all of them are fine, | ||
but be aware of the last one of the caveats below if you are not using the Gradle method: | ||
|
||
* If you are in a `sh` derivate like e.g. `bash` and Kotlin is installed and | ||
available in the `PATH`, you can just call the `*.main.kts` script like any | ||
other shell script: | ||
+ | ||
[source,bash] | ||
---- | ||
$ ./release.main.kts | ||
---- | ||
|
||
* If Kotlin is installed somewhere you can call it with the `*.main.kts` script | ||
as argument: | ||
+ | ||
[source,bash] | ||
---- | ||
$ path/to/kotlin release.main.kts | ||
---- | ||
|
||
* From the IDE you can create a run configuration that executes the `*.main.kts` script. | ||
|
||
* There is a Gradle task `preprocessWorkflows` that generates all YAML files from the | ||
according `*.main.kts` files. Additionally, there is also one task per workflow to | ||
only generate that one: | ||
+ | ||
[source,bash] | ||
---- | ||
$ ./gradlew preprocessReleaseWorkflow | ||
$ ./gradlew preprocessWorkflows | ||
---- | ||
|
||
|
||
|
||
== Caveats | ||
|
||
There are currently three known caveats with the approach we follow. | ||
|
||
* https://youtrack.jetbrains.com/issue/KTIJ-16532 | ||
+ | ||
If you navigate to a file in the dependencies, only a decompiled file is opened, | ||
even though the source JAR would be available. Also the quick documentation is missing. | ||
+ | ||
This can easily by mitigated by attaching the library to the normal project | ||
dependencies while having the need to navigate the source files or while editing them, | ||
which makes them properly viewable and documentation displayable in the editor. | ||
|
||
* https://youtrack.jetbrains.com/issue/KTIJ-14580 | ||
+ | ||
We use `@file:Import` to reduce code duplication by having common code in a common file. | ||
Unfortunately, this triggers a Kotlin IntelliJ plugin bug where the imported file cannot | ||
be loaded properly and so the things supplied by it like dependencies or common functions | ||
are not available. This makes most of the workflow `*.main.kts` files red as hell in the | ||
IDE currently. | ||
+ | ||
To reduce risk for eye-cancer while reading the `*.main.kts` scripts or to be able to | ||
sanely edit them, temporarily add the `@file:DependsOn` from the imported file to the | ||
importing file and wait a second, then remove the line again once you are done. | ||
|
||
* https://youtrack.jetbrains.com/issue/KT-42101 | ||
+ | ||
We use `@file:Import` to reduce code duplication by having common code in a common file. | ||
Unfortunately, this triggers a Kotlin bug where the compilation cache becomes confused | ||
if the imported file is changed without the importing file being changed too. | ||
+ | ||
If only the imported file is changed, it could happen that an old version is used, | ||
or it could also happen that classes added by a `@file:DependsOn` in the imported file | ||
are not available to the importing file. So if there was a change in the imported file, | ||
you either need to also change the importing file, or to properly execute the script, | ||
you need to delete the stale entry from the compilation cache which can be found at for example | ||
`~/.cache/main.kts.compiled.cache/` on Linux and `%LOCALAPPDATA%\main.kts.compiled.cache\` | ||
on Windows. Alternatively, you can also delete the whole cache directory. | ||
+ | ||
Another option is to disable the compilation cache for the execution by setting the | ||
environment variable `KOTLIN_MAIN_KTS_COMPILED_SCRIPTS_CACHE_DIR` or the system property | ||
`kotlin.main.kts.compiled.scripts.cache.dir` to an empty value, depending on the run | ||
method you chose. The Gradle tasks already do that, so when using the Gradle tasks you | ||
do not have this problem and it just works. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
#!/usr/bin/env kotlin | ||
|
||
/* | ||
* Copyright 2023 the original author or authors. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* https://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
@file:Import("common.main.kts") | ||
@file:Repository("https://bindings.krzeminski.it/") | ||
@file:DependsOn("actions:checkout:v4") | ||
@file:DependsOn("codecov:codecov-action:v5") | ||
|
||
import io.github.typesafegithub.workflows.actions.actions.Checkout | ||
import io.github.typesafegithub.workflows.actions.actions.Checkout.FetchDepth | ||
import io.github.typesafegithub.workflows.actions.codecov.CodecovAction | ||
import io.github.typesafegithub.workflows.domain.Concurrency | ||
import io.github.typesafegithub.workflows.domain.RunnerType | ||
import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest | ||
import io.github.typesafegithub.workflows.domain.triggers.MergeGroup | ||
import io.github.typesafegithub.workflows.domain.triggers.PullRequest | ||
import io.github.typesafegithub.workflows.domain.triggers.Push | ||
import io.github.typesafegithub.workflows.dsl.expressions.Contexts.github | ||
import io.github.typesafegithub.workflows.dsl.expressions.expr | ||
import io.github.typesafegithub.workflows.dsl.workflow | ||
|
||
workflow( | ||
name = "Verify Branches and PRs", | ||
on = listOf( | ||
Push( | ||
branchesIgnore = listOf( | ||
"master", | ||
"gh-pages" | ||
) | ||
), | ||
PullRequest(), | ||
MergeGroup() | ||
), | ||
sourceFile = __FILE__, | ||
targetFileName = "${__FILE__.name.substringBeforeLast(".main.kts")}.yml", | ||
// https://stackoverflow.com/a/72408109/16358266 | ||
concurrency = Concurrency( | ||
group = "${expr { github.workflow }}-${expr("${github.eventPullRequest.pull_request.number} || ${github.ref}")}", | ||
cancelInProgress = true | ||
) | ||
) { | ||
job( | ||
id = "check_all_workflow_yaml_consistency", | ||
name = "Check all Workflow YAML Consistency", | ||
runsOn = UbuntuLatest | ||
) { | ||
uses( | ||
name = "Checkout Repository", | ||
action = Checkout() | ||
) | ||
run( | ||
name = "Regenerate all Workflow YAMLs", | ||
command = """find .github/workflows -mindepth 1 -maxdepth 1 -name '*.main.kts' -exec {} \;""" | ||
) | ||
run( | ||
name = "Check for Modifications", | ||
command = """ | ||
git add --intent-to-add . | ||
git diff --exit-code | ||
""".trimIndent() | ||
) | ||
} | ||
|
||
job( | ||
id = "build-and-verify", | ||
name = "Build and Verify", | ||
runsOn = RunnerType.Custom(expr(Matrix.operatingSystem)), | ||
strategy = Strategy( | ||
matrix = Matrix.full | ||
) | ||
) { | ||
uses( | ||
name = "Checkout Repository", | ||
action = Checkout( | ||
// Codecov needs fetch-depth > 1 | ||
fetchDepth = FetchDepth.Value(2) | ||
) | ||
) | ||
uses( | ||
name = "Set up JDKs", | ||
action = SetupBuildEnv( | ||
additionalJavaVersion = expr(Matrix.javaVersion) | ||
) | ||
) | ||
run( | ||
name = "Build Spock", | ||
command = listOf( | ||
"./gradlew", | ||
"--stacktrace", | ||
"ghActionsBuild", | ||
""""-Dvariant=${expr(Matrix.variant)}"""", | ||
""""-DjavaVersion=${expr(Matrix.javaVersion)}"""" | ||
).joinToString(" "), | ||
// secrets are not injected for pull requests | ||
env = commonCredentials | ||
) | ||
uses( | ||
name = "Upload to Codecov.io", | ||
action = CodecovAction() | ||
) | ||
} | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we keep the
.yml
extension for the generated files, so that we can actually see the diff?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you can manually configure the target file name.
I changed it to
...yml
, but maybe we want to return to the default after you verified the changes to not have the unnecessary configuration?