Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write GitHub Actions workflow files in Kotlin instead of YAML #1630

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b6e5d76
Write GitHub Actions workflow files in Kotlin instead of Yaml
Vampire Apr 12, 2023
d22743b
Update to latest github-workflows-kt version and some improvements
Vampire Apr 17, 2023
b61b0eb
Move preprocessWorkflow task creation to build-logic
Vampire Apr 17, 2023
bf8ef4e
Verify consistency of all workflow YAMLs on pull requests and branches
Vampire Apr 17, 2023
bf79409
yaml => yml
Vampire Apr 17, 2023
d4f1d72
Use programmatic logic for variants and java versions
Vampire Apr 18, 2023
690d1a5
Use one central location to define the java versions and variants
Vampire Apr 19, 2023
0247fe2
Improve all script consistency check
Vampire Apr 19, 2023
955f8cb
Update github-workflows-kt to v0.44.0
Vampire Apr 28, 2023
8606f7b
Review feedback
Vampire Jun 1, 2023
170892d
Review feedback v2
Vampire Jun 9, 2023
fc23128
Catch up changes from master
Vampire Aug 15, 2023
7049f8d
Update github-workflows-kt version to 0.50.0
Vampire Aug 22, 2023
93aa39e
Set group for GHA tasks to 'github actions'
Vampire Aug 22, 2023
f2e5f71
Catch up changes from master
Vampire Sep 18, 2023
6119c3f
Catch up changes from master
Vampire Oct 11, 2023
04d9237
Update github-workflows-kt version to 1.3.0
Vampire Oct 13, 2023
c3c1406
Update github-workflows-kt version to 1.4.0
Vampire Oct 31, 2023
23b171a
Catch up changes from master
Vampire Oct 31, 2023
80aae4f
Update github-workflows-kt version to 1.12.0
Vampire Mar 6, 2024
4e02013
Catch up changes from master
Vampire Mar 6, 2024
17bff51
Update Kotlin code style settings
Vampire Mar 6, 2024
13a6041
Catch up changes from master
Vampire Mar 18, 2024
c2c8131
Catch up changes from master
Vampire Apr 20, 2024
d8ea1ce
Catch up changes from master
Vampire May 16, 2024
267d269
Only use zulu on arm macOS
Vampire May 16, 2024
c02372f
Update github-workflows-kt version to 2.0.0
Vampire May 22, 2024
1520e8d
Update github-workflows-kt version to 2.3.0 and Maven-based bindings
Vampire Aug 4, 2024
2ffa401
Fix typo in workflow readme
Vampire Aug 8, 2024
9ad73eb
Improve all-workflows consistency check
Vampire Aug 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,10 @@ indent_size = 2
# The file contains important whitespace at the end of the line in a multi-line string.
# and editorconfig doesn't seem to respect multi-line strings.
trim_trailing_whitespace = false

[*.{kt,kts}]
indent_size = 4
ij_kotlin_allow_trailing_comma = false
ij_kotlin_allow_trailing_comma_on_call_site = false
ktlint_code_style = intellij_idea
ktlint_standard_function-signature = disabled
2 changes: 1 addition & 1 deletion .github/actions/setup-jdks/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ runs:
uses: actions/setup-java@v4
with:
# Temurin JDK 8 for macos on ARM is not available: https://github.com/adoptium/adoptium/issues/96
distribution: ${{ runner.os == 'macOS' && 'zulu' || 'temurin' }}
distribution: ${{ ((runner.os == 'macOS') && (runner.arch == 'ARM64')) && 'zulu' || 'temurin' }}
java-version: 8
- name: Prepare JDK8 env var
shell: bash
Expand Down
98 changes: 98 additions & 0 deletions .github/workflows/README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
== The YAML workflow files vs. the `*.main.kts` files

The YAML workflow files are generated from the `*.main.kts` files.

These use the https://github.com/typesafegithub/github-workflows-kt[github-workflows-kt]
Kotlin DSL library to conveniently and type-safely write GitHub Action workflow files.

As there is no official built-in support in GitHub Actions yet until
https://github.com/orgs/community/discussions/15904 is considered, the YAML files
need to be generated manually.

There is a safeguard check in all the generated files that this is not forgotten.
Running a workflow where the according `*.main.kts` produces a different output will
fail the execution. Additionally, the workflow that runs for pull requests checks
the consistency of all the YAML files as not all are run for pull requests.



== Ways to generate the YAML workflow files

There are multiple ways to generate the YAML files and all of them are fine,
but be aware of the last one of the caveats below if you are not using the Gradle method:

* If you are in a `sh` derivate like e.g. `bash` and Kotlin is installed and
available in the `PATH`, you can just call the `*.main.kts` script like any
other shell script:
+
[source,bash]
----
$ ./release.main.kts
----

* If Kotlin is installed somewhere you can call it with the `*.main.kts` script
as argument:
+
[source,bash]
----
$ path/to/kotlin release.main.kts
----

* From the IDE you can create a run configuration that executes the `*.main.kts` script.

* There is a Gradle task `preprocessWorkflows` that generates all YAML files from the
according `*.main.kts` files. Additionally, there is also one task per workflow to
only generate that one:
+
[source,bash]
----
$ ./gradlew preprocessReleaseWorkflow
$ ./gradlew preprocessWorkflows
----



== Caveats

There are currently three known caveats with the approach we follow.

* https://youtrack.jetbrains.com/issue/KTIJ-16532
+
If you navigate to a file in the dependencies, only a decompiled file is opened,
even though the source JAR would be available. Also the quick documentation is missing.
+
This can easily by mitigated by attaching the library to the normal project
dependencies while having the need to navigate the source files or while editing them,
which makes them properly viewable and documentation displayable in the editor.

* https://youtrack.jetbrains.com/issue/KTIJ-14580
+
We use `@file:Import` to reduce code duplication by having common code in a common file.
Unfortunately, this triggers a Kotlin IntelliJ plugin bug where the imported file cannot
be loaded properly and so the things supplied by it like dependencies or common functions
are not available. This makes most of the workflow `*.main.kts` files red as hell in the
IDE currently.
+
To reduce risk for eye-cancer while reading the `*.main.kts` scripts or to be able to
sanely edit them, temporarily add the `@file:DependsOn` from the imported file to the
importing file and wait a second, then remove the line again once you are done.

* https://youtrack.jetbrains.com/issue/KT-42101
+
We use `@file:Import` to reduce code duplication by having common code in a common file.
Unfortunately, this triggers a Kotlin bug where the compilation cache becomes confused
if the imported file is changed without the importing file being changed too.
+
If only the imported file is changed, it could happen that an old version is used,
or it could also happen that classes added by a `@file:DependsOn` in the imported file
are not available to the importing file. So if there was a change in the imported file,
you either need to also change the importing file, or to properly execute the script,
you need to delete the stale entry from the compilation cache which can be found at for example
`~/.cache/main.kts.compiled.cache/` on Linux and `%LOCALAPPDATA%\main.kts.compiled.cache\`
on Windows. Alternatively, you can also delete the whole cache directory.
+
Another option is to disable the compilation cache for the execution by setting the
environment variable `KOTLIN_MAIN_KTS_COMPILED_SCRIPTS_CACHE_DIR` or the system property
`kotlin.main.kts.compiled.scripts.cache.dir` to an empty value, depending on the run
method you chose. The Gradle tasks already do that, so when using the Gradle tasks you
do not have this problem and it just works.
117 changes: 117 additions & 0 deletions .github/workflows/branches-and-prs.main.kts
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the .yml extension for the generated files, so that we can actually see the diff?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you can manually configure the target file name.
I changed it to ...yml, but maybe we want to return to the default after you verified the changes to not have the unnecessary configuration?

Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
#!/usr/bin/env kotlin

/*
* Copyright 2023 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

@file:Import("common.main.kts")
@file:Repository("https://bindings.krzeminski.it/")
@file:DependsOn("actions:checkout:v4")
@file:DependsOn("codecov:codecov-action:v4")

import io.github.typesafegithub.workflows.actions.actions.Checkout
import io.github.typesafegithub.workflows.actions.actions.Checkout.FetchDepth
import io.github.typesafegithub.workflows.actions.codecov.CodecovAction
import io.github.typesafegithub.workflows.domain.Concurrency
import io.github.typesafegithub.workflows.domain.RunnerType
import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest
import io.github.typesafegithub.workflows.domain.triggers.MergeGroup
import io.github.typesafegithub.workflows.domain.triggers.PullRequest
import io.github.typesafegithub.workflows.domain.triggers.Push
import io.github.typesafegithub.workflows.dsl.expressions.Contexts.github
import io.github.typesafegithub.workflows.dsl.expressions.expr
import io.github.typesafegithub.workflows.dsl.workflow

workflow(
name = "Verify Branches and PRs",
on = listOf(
Push(
branchesIgnore = listOf(
"master",
"gh-pages"
)
),
PullRequest(),
MergeGroup()
),
sourceFile = __FILE__,
targetFileName = "${__FILE__.name.substringBeforeLast(".main.kts")}.yml",
// https://stackoverflow.com/a/72408109/16358266
concurrency = Concurrency(
group = "${expr { github.workflow }}-${expr("${github.eventPullRequest.pull_request.number} || ${github.ref}")}",
cancelInProgress = true
)
) {
job(
id = "check_all_workflow_yaml_consistency",
name = "Check all Workflow YAML Consistency",
runsOn = UbuntuLatest
) {
uses(
name = "Checkout Repository",
action = Checkout()
)
run(
name = "Regenerate all Workflow YAMLs",
command = """find .github/workflows -mindepth 1 -maxdepth 1 -name '*.main.kts' -exec {} \;"""
)
run(
name = "Check for Modifications",
command = """
git add --intent-to-add .
git diff --exit-code
""".trimIndent()
)
}

job(
id = "build-and-verify",
name = "Build and Verify",
runsOn = RunnerType.Custom(expr(Matrix.operatingSystem)),
strategy = Strategy(
matrix = Matrix.full
)
) {
uses(
name = "Checkout Repository",
action = Checkout(
// Codecov needs fetch-depth > 1
fetchDepth = FetchDepth.Value(2)
)
)
uses(
name = "Set up JDKs",
action = SetupBuildEnv(
additionalJavaVersion = expr(Matrix.javaVersion)
)
)
run(
name = "Build Spock",
command = listOf(
"./gradlew",
"--stacktrace",
"ghActionsBuild",
""""-Dvariant=${expr(Matrix.variant)}"""",
""""-DjavaVersion=${expr(Matrix.javaVersion)}""""
).joinToString(" "),
// secrets are not injected for pull requests
env = commonCredentials
)
uses(
name = "Upload to Codecov.io",
action = CodecovAction()
)
}
}
Loading
Loading