Skip to content

Commit

Permalink
Review feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
Vampire committed Jun 6, 2023
1 parent 24a9b9c commit 69bfb43
Show file tree
Hide file tree
Showing 11 changed files with 361 additions and 198 deletions.
91 changes: 91 additions & 0 deletions .github/workflows/README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
== The YAML workflow files vs. the `*.main.kts` files

The YAML workflow files are generated from the `*.main.kts` files.

These use the https://github.com/typesafegithub/github-workflows-kt[github-workflows-kt]
Kotlin DSL library to conveniently and type-safely write GitHub Action workflow files.

As there is no official built-in support in GitHub Actions yet until
https://github.com/orgs/community/discussions/15904 is considered, the YAML files
need to be generated manually.

There is a safeguard check in all the generated files that this is not forgotten.
Running a workflow where the according `*.main.kts` produces a different output will
fail the execution. Additionally, the workflow that runs for pull requests checks
the consistency of all the YAML files as not all are run for pull requests.



== Ways to generate the YAML workflow files

There are multiple ways to generate the YAML files and all of them are fine:

* If you are in a `sh` derivate like e.g. `bash` and Kotlin is installed and
available in the `PATH`, you can just call the `*.main.kts` script like any
other shell script:
+
[source,bash]
----
$ ./release.main.kts
----

* If Kotlin is installed somewhere you can call it with the `*.main.kts` script
as argument:
+
[source,bash]
----
$ path/to/kotlin release.main.kts
----

* From the IDE you can create a run configuration that executes the `*.main.kts` script.

* There is a Gradle task `preprocessWorkflows` that generates all YAML files from the
according `*.main.kts` files. Additionally, there is also one task per workflow to
only generate that one:
+
[source,bash]
----
$ ./gradlew preprocessReleaseWorkflow
$ ./gradlew preprocessWorkflows
----



== Caveats

There are currently three known caveats with the approach we follow.

* https://youtrack.jetbrains.com/issue/KTIJ-16532
+
If you navigate to a file in the dependencies, only a decompiled file is opened,
even though the source JAR would be available. Also the quick documentation is missing.
+
This can easily by mitigated by attaching the library to the normal project
dependencies while having the need to navigate the source files or while editing them,
which makes them properly viewable and documentation displayable in the editor.

* https://youtrack.jetbrains.com/issue/KTIJ-14580
+
We use `@file:Import` to reduce code duplication by having common code in a common file.
Unfortunately, this triggers a Kotlin IntelliJ plugin bug where the imported file cannot
be loaded properly and so the things supplied by it like dependencies or common functions
are not available. This makes most of the workflow `*.main.kts` files red as hell in the
IDE currently.
+
To reduce risk for eye-cancer while reading the `*.main.kts` scripts or to be able to
sanely edit them, temporarily add the `@file:DependsOn` from the imported file to the
importing file and wait a second, then remove the line again once you are done.

* https://youtrack.jetbrains.com/issue/KT-42101
+
We use `@file:Import` to reduce code duplication by having common code in a common file.
Unfortunately, this triggers a Kotlin bug where the compilation cache becomes confused
if the imported file is changed without the importing file being changed too.
+
If only the imported file is changed, it could happen that a old version is used,
or it could also happen that classes added by a `@file:DependsOn` in the imported file
are not available to the importing file. So if there was a change in the imported file,
you either need to also change the importing file, or to properly execute the script,
you need to delete the stale entry from the compilation cache which can be found at for example
`~/.cache/main.kts.compiled.cache/` on Linux and `%LOCALAPPDATA%\main.kts.compiled.cache\`
on Windows. Alternatively, you can also delete the whole cache directory.
83 changes: 10 additions & 73 deletions .github/workflows/branches-and-prs.main.kts
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,17 @@
* limitations under the License.
*/

@file:DependsOn("io.github.typesafegithub:github-workflows-kt:0.44.0")
@file:DependsOn("org.codehaus.groovy:groovy:3.0.15")
@file:Import("common.main.kts")

import groovy.lang.Binding
import groovy.lang.GroovyShell
import io.github.typesafegithub.workflows.actions.actions.CheckoutV3
import io.github.typesafegithub.workflows.actions.codecov.CodecovActionV3
import io.github.typesafegithub.workflows.actions.gradle.GradleBuildActionV2
import io.github.typesafegithub.workflows.domain.Concurrency
import io.github.typesafegithub.workflows.domain.RunnerType
import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest
import io.github.typesafegithub.workflows.domain.actions.Action.Outputs
import io.github.typesafegithub.workflows.domain.actions.LocalAction
import io.github.typesafegithub.workflows.domain.triggers.PullRequest
import io.github.typesafegithub.workflows.domain.triggers.Push
import io.github.typesafegithub.workflows.dsl.expressions.Contexts
import io.github.typesafegithub.workflows.dsl.expressions.Contexts.github
import io.github.typesafegithub.workflows.dsl.expressions.expr
import io.github.typesafegithub.workflows.dsl.workflow
import io.github.typesafegithub.workflows.yaml.writeToFile
Expand All @@ -53,7 +48,7 @@ workflow(
targetFileName = "${__FILE__.name.substringBeforeLast(".main.kts")}.yml",
// https://stackoverflow.com/a/72408109/16358266
concurrency = Concurrency(
group = "${expr { github.workflow }}-${expr("${Contexts.github.eventPullRequest.pull_request.number} || ${Contexts.github.ref}")}",
group = "${expr { github.workflow }}-${expr("${github.eventPullRequest.pull_request.number} || ${github.ref}")}",
cancelInProgress = true
)
) {
Expand All @@ -72,38 +67,12 @@ workflow(
)
}

val (javaVersions, variants) = getMatrixAxes()
job(
id = "build-and-verify",
name = "Build and Verify",
runsOn = RunnerType.Custom(expr("matrix.os")),
_customArguments = mapOf(
"strategy" to mapOf(
"fail-fast" to false,
"matrix" to mapOf(
"os" to listOf("ubuntu-latest"),
"variant" to variants,
"java" to javaVersions,
"exclude" to javaVersions
.filter { it.toInt() >= 17 }
.map { javaVersion ->
mapOf(
"os" to "ubuntu-latest",
"variant" to "2.5",
"java" to javaVersion
)
},
"include" to listOf("windows-latest", "macos-latest")
.flatMap { os -> variants.map { os to it } }
.map { (os, variant) ->
mapOf(
"os" to os,
"variant" to variant,
"java" to javaVersions.first()
)
}
)
)
runsOn = RunnerType.Custom(expr(Matrix.operatingSystem)),
strategy = Strategy(
matrix = Matrix.full
)
) {
uses(
Expand All @@ -116,58 +85,26 @@ workflow(
uses(
name = "Set up JDKs",
action = SetupBuildEnv(
additionalJavaVersion = expr("matrix.java")
additionalJavaVersion = expr(Matrix.java)
)
)
val SPOCK_BUILD_CACHE_USERNAME by Contexts.secrets
val SPOCK_BUILD_CACHE_PASSWORD by Contexts.secrets
val GRADLE_ENTERPRISE_ACCESS_KEY by Contexts.secrets
uses(
name = "Build Spock",
action = GradleBuildActionV2(
arguments = listOf(
"--no-parallel",
"--stacktrace",
"ghActionsBuild",
""""-Dvariant=${expr("matrix.variant")}"""",
""""-DjavaVersion=${expr("matrix.java")}""""
""""-Dvariant=${expr(Matrix.variant)}"""",
""""-DjavaVersion=${expr(Matrix.java)}""""
).joinToString(" ")
),
// secrets are not injected for pull requests
env = linkedMapOf(
"ORG_GRADLE_PROJECT_spockBuildCacheUsername" to expr(SPOCK_BUILD_CACHE_USERNAME),
"ORG_GRADLE_PROJECT_spockBuildCachePassword" to expr(SPOCK_BUILD_CACHE_PASSWORD),
"GRADLE_ENTERPRISE_ACCESS_KEY" to expr(GRADLE_ENTERPRISE_ACCESS_KEY)
)
env = commonCredentials
)
uses(
name = "Upload to Codecov.io",
action = CodecovActionV3()
)
}
}.writeToFile()

data class SetupBuildEnv(
val additionalJavaVersion: String? = null
) : LocalAction<Outputs>("./.github/actions/setup-build-env") {
override fun toYamlArguments() =
additionalJavaVersion?.let { linkedMapOf("additional-java-version" to it) } ?: linkedMapOf()

override fun buildOutputObject(stepId: String): Outputs = Outputs(stepId)
}

fun getMatrixAxes(): Pair<List<String>, List<String>> {
val binding = object : Binding() {
lateinit var javaVersions: List<String>
lateinit var variants: List<String>

override fun setVariable(name: String?, value: Any?) {
when (name) {
"javaVersions" -> javaVersions = (value as List<Int>).map { it.toString() }
"variants" -> variants = value as List<String>
}
}
}
GroovyShell(binding).evaluate(__FILE__.parentFile.resolve("../../matrix.groovy"))
return binding.javaVersions to binding.variants
}
24 changes: 5 additions & 19 deletions .github/workflows/codeql-analysis.main.kts
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,14 @@
* limitations under the License.
*/

@file:DependsOn("io.github.typesafegithub:github-workflows-kt:0.44.0")
@file:Import("common.main.kts")

import io.github.typesafegithub.workflows.actions.actions.CheckoutV3
import io.github.typesafegithub.workflows.actions.github.CodeqlActionAnalyzeV2
import io.github.typesafegithub.workflows.actions.github.CodeqlActionInitV2
import io.github.typesafegithub.workflows.actions.gradle.GradleBuildActionV2
import io.github.typesafegithub.workflows.domain.Concurrency
import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest
import io.github.typesafegithub.workflows.domain.actions.Action
import io.github.typesafegithub.workflows.domain.actions.LocalAction
import io.github.typesafegithub.workflows.domain.triggers.Cron
import io.github.typesafegithub.workflows.domain.triggers.PullRequest
import io.github.typesafegithub.workflows.domain.triggers.Push
Expand Down Expand Up @@ -65,12 +63,9 @@ workflow(
name = "CodeQL-Build",
// CodeQL runs on UbuntuLatest, WindowsLatest, and MacOSLatest
runsOn = UbuntuLatest,
_customArguments = mapOf(
"strategy" to mapOf(
"fail-fast" to false,
"matrix" to mapOf(
"variant" to listOf("2.5", "3.0", "4.0")
)
strategy = Strategy(
matrix = Matrix(
variants = Matrix.axes.variants
)
)
) {
Expand Down Expand Up @@ -122,7 +117,7 @@ workflow(
"--stacktrace",
"--no-build-cache",
"testClasses",
""""-Dvariant=${expr("matrix.variant")}""""
""""-Dvariant=${expr(Matrix.variant)}""""
).joinToString(" ")
)
)
Expand All @@ -132,12 +127,3 @@ workflow(
)
}
}.writeToFile()

data class SetupBuildEnv(
val additionalJavaVersion: String? = null
) : LocalAction<Action.Outputs>("./.github/actions/setup-build-env") {
override fun toYamlArguments() =
additionalJavaVersion?.let { linkedMapOf("additional-java-version" to it) } ?: linkedMapOf()

override fun buildOutputObject(stepId: String): Outputs = Outputs(stepId)
}
Loading

0 comments on commit 69bfb43

Please sign in to comment.